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eStream Application Install Manager Low Level Design 



Functionality 

The Application Install Manager (AIM) is a component of the eStream client executable. 
It is responsible for installing and uninstalling eStream applications at the request of the 
License Subscription Manager (LSM). AIM uses the information contained in an Appln- 
stallBlock to prepare the user's system for execution of a given eStream application. It 
creates registry entries, copies files, and updates the file spoofing database. The user can 
then launch his application via a local shortcut or a shortcut on the eStream drive. Unin- 
stallation involves undoing all changes made to the user's system by AIM during installa- 
tion. 

Data type definitions 

This component uses the AppInstallBlock, but doesn't define it. This is defined in a low- 
level design document for the Builder component. 

The AppInstallBlock is a binary data file with a versioned interface, basically consisting 
of: 

• a header 

• a list of files to install or send to the file spoofer 

• a list of registry entries to install or remove 

• a set of prefetch requests to communicate to the profile/prefetch component 

• a set of initial profile data to communicate to the profile/prefetch component 
(post-version 1.0) 

• a comment section 

• an embedded DLL that can be loaded and executed for custom install needs 

• a section containing a license agreement to be shown to the user 

Many of the AIMsc functions take an ABFileRef as an argument, which is an opaque 
pointer to the following structure: 
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HANDLE 

AIBFileHeader 
AIBIndexEntry 
LPCTSTR 



FileHandle; 
FileHeader ; 
*IndexEntries; 
AppName ; 



Omnishift Technologies, Inc. 



Company Confidential 



eStream <COMPONENT> Low Level Design 

} AIBFilelnfo, *pAIBFileInf o; 

It is assumed that an external header file will be available that defines structures such as 
AffiFileHeader and AIBIndexEntry. For now, refer to the AppInstallBlock-LLD for how 
they might be defined. 

Also, each application has a prefetch data file created for it an install time that is initial- 
ized with prefetch data from the AppfristallBlock. This data file is named and located as 
described in the Component Design section, and just consists of a non-padded list of the 
following structures: 

typedef struct 
{ 

UINT32 FileNumber; 
UINT32 BlockNumber; 

} Prefetchltem, *pPref etchltem; 
The following data types are used in the AIM and AIMsc interfaces: 

typedef void *AIBFileRef; 
Error codes that are assumed to be defined somewhere are: 

SUCCESS (0) 

ERROR BUFFER TOO SMALL 



Interface definitions 
Application installation/uninstallation 

There are only two functions exposed by AIM, one for application installation, and an- 
other for application uninstallation. Only the License Subscription Manager will be call- 
ing these functions. 

UINT32 

AIMInstallApplication(UINT8Appld[16], LPCTSTR PathToAlB) 
Parameters 

Appld 

[in] The application ID of the eStream application to install. 
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PathToAIB 

[in] Pointer to a null-terminated string that specifies a path to an Appln- 
stallBlock file to install. 

Return Values 

SUCCESS (0) if all the actions specified in the AppInstallBlock were performed 
successfully, an error code otherwise. 

Comments 

None. 

UINT32 

AIMUninstallApplication(UINT8 Appld[16]) 

Parameters 

Appld 

[in] The application ID of an existing eStream application to uninstall. 
Return Values 

If the specified application ID is not recognized, or the original AppInstallBlock 
is not found, an error code will be returned. Otherwise, AIM will make an attempt 
to undo all of the actions it performed while installing this application. It will re- 
turn SUCCESS (0) if it undid enough of these actions so that any future installa- 
tion of the same application will succeed. 

Comments 

None. 

AIM Sub-Component Interface 

Much of the functionality required by the AIM design will be useful to the Builder testing 
framework as well. This functionality will be treated as a sub-component within the AIM 
component, called AIMsc, and will export a well-defined interface. That interface is de- 
fined as follows. 

UINT32 

AIMscOpenApplnstallBlock(LPCTSTR PathToAIB, AlBFileRef *pAIBFile) 
Parameters 
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PathToAIB 

[in] Pointer to a null-terminated string that specifies a path to an Appln- 
stallBlock file to open. 

pAIBFile 

[out] Returns a reference to an open AppInstallBlock file. 
Return Values 

SUCCESS (0) if the AppInstallBlock was opened successfully and validated, an 
error code otherwise. 

Comments 

The reference returned by this function can be used as a parameter to any of the 
other functions that take an AIBFileRef 

UINT32 

AIMscCloseApplnstallBlock(AIBFileRef AIBFile) 
Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

Return Values 

SUCCESS (0) if the close succeeded, an error code otherwise. 
Comments 
None. 

void 

AIMscGetAIBVersion(AIBFileRef AIBFile, UINT32 *pAIBVersion) 
Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AMscOpenAppInstallBlock. 
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pAIB Version 

[out] Returns the value of the AibVersion field in the AppInstallBlock. 
Return Values 

SUCCESS (0) if the value was successfully retrieved, an error code otherwise. 
Comments 
None. 

void 

AIMscGetAIBAppld(AIBFileRef AlBFile, UINT8 pAIBAppld[16]) 
Parameters 

AlBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

pAIB Version 

[out] Returns the value of the Appld field in the AppInstallBlock. 
Return Values 

SUCCESS (0) if the value was successfully retrieved, an error code otherwise. 
Comments 
None. 

void 

AIMscGetAIBVersionNo(AIBFileRef AlBFile, UINT32 *pAIBVersionNo) 
Parameters 

AlBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

pAIBVersionNo 
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[out] Returns the value of the VersionNo field in the AppInstallBlock. 
Return Values 

SUCCESS (0) if the value was successfully retrieved, an error code otherwise. 
Comments 
None. 

void 

AIMscGetAIBShouldRebootf 
AlBFileRef AlBFile, 
BOOLEAN *pAIBShouldReboot) 

Parameters 

AlBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

pAIBShouldReboot 

[out] Returns the value of the ShouldReboot flag in the AppInstallBlock. 
Return Values 

SUCCESS (0) if the value was successfully retrieved, an error code otherwise. 
Comments 

None. 
UINT32 

AIMscGetAIBAppNamef 
AlBFileRef AlBFile, 
LPTSTR pAIBAppName, 
UINT16 *pSizeAIBAppName) 

Parameters 

AlBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 
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pAIBAppName 

[out] The value of the ApplicationName field in the AppInstallBlock is 
copied into the memory pointed to by this address (it will be null termi- 
nated). 

pSizeAIBAppName 

[in, out] On input, should point to the size of the memory at pAIBApp- 
Name. On output, will point to the total bytes needed to hold the entire 
string if ERRORBUFFERTOOSMALL is returned, otherwise is unde- 
fined. 

Return Values 

SUCCESS (0) if the value was successfully retrieved, ER- 
RORBUFFERTOOSMALL if the buffer is too small to hold the entire string, 
or another error code otherwise. 

Comments 

None. 

UINT32 

AlMscCheckAIBCompatibleOSf 
AlBFileRef AlBFile, 
BOOLEAN *pWasOSCompatible) 

Parameters 

AlBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

pWasOSCompatible 

[out] Returns TRUE if the AppInstallBlock can be installed on the current 
OS, FALSE otherwise. 

Return Values 

SUCCESS (0) if the OS version was successfully retrieved and checked, an error 
code otherwise. 

Comments 
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This function will check if the currently installed operating system and service is 
compatible with the specified AppInstallBlock (using the compatibility informa- 
tion contained in the AppInstallBlock). If not, it will display a detailed message to 
the user and return FALSE in pWasOSCompatible, otherwise it will do nothing 
and return TRUE in p WasOSCompatible. 

UINT32 

AIMsclnstallAppFilesf 

AlBFileRef AlBFile, 

HKEY SpoofKey, 

HKEY SpoofRefCountKey, 

LPCTSTR InstallLogFile, 

BOOLEAN *plsRebootNeeded) 

Parameters 

AlBFile 



[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

SpoofKey 

[in] An open handle to the registry key where file-spoofing data is stored. 
SpoofRefCountKey 

[in] An open handle to the registry key where file-spoofing reference 
counts are stored. 

InstallLogFile 



[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 



phRebootNeeded 



[out] Returns TRUE if a reboot is needed to complete the file copying, 
FALSE otherwise. 



Return Values 



SUCCESS (0) if all file install operations succeeded, an error code otherwise. 



Comments 
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This function will perform the file copies and add the file spoofing entries speci- 
fied in the File section of the AppInstallBlock. Changes will be appended to the 
install log file (it is created if it doesn't already exist). 

For the sake of getting an error back to the user as soon as possible, this function 
will not undo file copies or spoof entry additions if it fails. AIMscUninstal- 
lAppFiles should be called to do so after the caller informs the user of the error. 



UINT32 

AIMscUninstallAppFiles( 

AlBFileRef AlBFile, 

HKEY SpoofKey, 

HKEY SpoofRefCountKey, 

LPCTSTR InstallLogFile, 

BOOLEAN *plsRebootNeeded) 

Parameters 



AlBFile 



[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

SpoofKey 

[in] An open handle to the registry key where file-spoofing data is stored. 
SpoofRefCountKey 



[in] An open handle to the registry key where file-spoofing reference 
counts are stored. 



InstallLogFile 



[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 



pIsRebootNeeded 



[out] Returns TRUE if a reboot is needed to complete the file deletions, 
FALSE otherwise. 



Return Values 
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SUCCESS (0) if enough of the file install operations were reversed so that re- 
installation will succeed and so that the system is in a consistent state. Otherwise, 
an error code is returned. 

Comments 

This function will reverse the file additions and remove the file spoof database en- 
tries specified in the install log file. 

UINT32 

AIMsclnstallAppVariablesf 
AlBFileRef AlBFile, 
LPCTSTR InstallLogFile) 

Parameters 

AlBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

InstallLogFile 

[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 

Return Values 

SUCCESS (0) if all variable modifications succeeded, an error code otherwise. 
Comments 

This function will perform the add/remove variable (i.e. registry entry) changes 
specified in the Variable section of the AppInstallBlock. Changes will be ap- 
pended to the install log file (it is created if it doesn't already exist). 

For the sake of getting an error back to the user as soon as possible, this fiinction 
will not undo registry modifications if it fails. AIMscUninstallAppVariables 
should be called to do so after informing the user of the error. 

UINT32 

AIMscUninstallAppVariablesf 
AlBFileRef AlBFile, 
LPCTSTR InstallLogFile) 

Parameters 
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AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

InstallLogFile 

[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 

Return Values 

SUCCESS (0) if enough of the variable changes were reversed so that re- 
installation will succeed and so that the registry is in a consistent state. Otherwise, 
an error code is returned. 

Comments 

This function will reverse the add/remove variable (i.e. registry entry) changes 
specified in the install log file. 

UINT32 

AIMsclnstallAppPrefetchFile(AIBFileRef AIBFile, LPCTSTR PrefetchFile) 
Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

PrefetchFile 

[in] A null-terminated string representing the path to the prefetch file to be 
created. 

Return Values 

SUCCESS (0) if prefetch block installation succeeded, an error code otherwise. 
Comments 

This function will install the prefetch information contained in the Prefetch sec- 
tion of the AppInstallBlock into PrefetchFile. 
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UINT32 

AIMscUninstallAppPrefetchFile(AIBFileRefAIBFile, LPCTSTR PrefetchFile) 
Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

PrefetchFile 

[in] A null-terminated string representing the path to the prefetch file to be 
uninstalled. 

Return Values 

SUCCESS (0) if prefetch block uninstallation succeeded, an error code otherwise. 
Comments 

This function will remove the prefetch information stored at PrefetchFile. 
UINT32 

AIMsclnstallAppProfileFile(AIBFileRef AIBFile, LPCTSTR ProfileFile) 
UINT32 

AIMscUninstallAppProfileFile(AIBFileRef AIBFile, LPCTSTR ProfileFile) 

(NOT FUNCTIONAL IN ESTREAM 1.0) 

UINT32 

AIMscCallCustomlnstall(AIBFileRef AIBFile) 
Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

Return Values 

An error code if the custom code .dll could not be extracted, loaded and called. 
Otherwise, returns the value returned by the InstallQ function in the custom code 
dll. 
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Comments 

This function will extract and load the custom code .dll included in the Code sec 
tion of the AppInstallBlock, and then call the exported .dll function named In- 
stallQ. 

UINT32 

AIMscCallCustomUninstall(AIBFileRefAIBFile) 
Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

Return Values 

An error code if the custom code .dll could not be extracted, loaded and called. 
Otherwise, returns the value returned by the UninstallQ function in the custom 
code .dll. 

Comments 

This function will extract and load the custom code .dll included in the Code sec- 
tion of the AppInstallBlock, and then call the exported .dll function named 
UninstallQ. 

UINT32 

AIMscEnforceLicenseAgreementf 
AlBFileRef AIBFile, 
BOOLEAN *pBUserAgreed) 

Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

pB User Agreed 

[out] Returns TRUE if the user agreed to the license terms, FALSE other- 
wise. 

Return Values 
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SUCCESS (0) if the license agreement was successfully displayed, an error code 
otherwise. 



Comments 

This function will extract the license agreement text included in the LicenseA- 
greement section of the AppInstallBlock and display it to the user. The user will 
be given the option to agree or not agree to the license (probably via a pair of but- 
tons in a dialog). 

UINT32 

AIMscDisplayComment(AIBFileRefAIBFile) 
Parameters 



AIBFile 



[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

Return Values 



SUCCESS (0) if the comment was successfully displayed, an error code other- 
wise. 



Comments 



This function will display to the user the comment included in the Comment 
tion of the AppInstallBlock. 



Component design 

AIMsc does not have hard-coded knowledge regarding any of the standard registry and 
file locations used by AIM, which is why the functions in its interface take as inputs 
specifiers for filenames and base registry locations. Conversely, AIM itself has no 
knowledge of the internal structure of the AppInstallBlock file, which is why it must call 
AIMsc functions to work with such files. 

Expansion is perform on registry entries and file paths containing certain variables when 
they are read from the AppInstallBlock. These variables are defined in the Builder-LLD 
and will be recognized and expanded by AIM. (This includes file-spoof entries ) 
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AIM stores its data in the expected places for an eStream client component. All of the 
data AIM stores is user-specific, so it makes no use of the global locations defined for 
eStreams. 

Registry keys 

AIM stores its registry keys and values under: 

HKEY_CUl^NT_USER\SOITWARE\Onmisluft\eStream\AIM 

This key will have its permissions modified so that ordinary users cannot modify the key 
(but the eStream client service will be given privileges so that it can do so) Here are the 
subkeys AIM places under this key: 

"SpoofEntries" 

Spoof entries are placed here. All spoofing is done globally, so there is no need to place it 
under an eStream-app specific key. Each value under this key is a pair of pathnames as 
follows: 

<old-pathname> (REGSZ) 

- <spoofed-pathname> 

"SpoofEntriesRefCounts" 

Reference counting for spoof entries is done here. If multiple eStream apps are installed 
that want to spoof the same file, the entries must be ref-counted so that uninstall does not 
break the other apps. Each value under this key is a pair like this: 

<o!d-pathname> (REGDWORD) 

- <ref-count> 

Every value under SpoofEntries has a value under SpoofEntriesRefCounts with the same 
value name. 

"<AppId>" 



Every installed eStream app has its own subkey whose name is a string representation of 
its Appld, like so: "{00000000-0000-0000-0000-00000000000}". The values stored un- 
der each such key are: 

Appld (REGJBINARY) 

- Appld in binary form (16 bytes) 
AppName (REG SZ) 

- name of the application (same as in the AppInstallBlock) 
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AppInstallBlockPath (REG_SZ) 

- path to the AppInstallBlock for the application 
AppInstallState (REGDWORD) 

- a value of 0 means app is installed, 1 means install is in progress, 2 mean 
uninstall is in progress. 

Files 

AIM stores per-user files at the following path: 

(Path to the user's home directory)\Application Data\Omnishift\eStream\AIM 

For each installed application, a separate data folder is created. The name of the folder is 
the Appld of the application in GU1D ASCII format, like so: "{00000000-0000-0000- 
0000-00000000000}". The files stored under each such folder are: 

<GUID string>-AIB.dat - the AppInstallBlock file for the application 

<GUID string>-Prefetch.dat - the prefetch data file for the application 
InstallLogtxt - a generated log of what to do during uninstall 

The Prefetch data file is simply an array of Prefetchltem structures (as described in the 
Data Structures section). 

The InstallLogtxt is a list of undoable actions taken during installation. This log will be 
used during uninstall to determine which files and entries are safe to remove. Each line in 
the file contains one change, and is of the form: 

ADDED or OVERWROTE or SPOOFED FILE "<filename>" (fully qualified) 
ADDED or OVERWROTE KEY "<keyname>" (fully qualified) 
ADDED or OVERWROTE VALUE "<valuename>" (fully qualified) 

AIMInstallApplication Prototype 

Installing an eStream application consists of the following steps: 

1. Preparing for the installation 

2. Displaying a license agreement to the user and having him agree to it 

3. Installing all required local files and spoof entries for this app 

4. Setting/removing registry entries as required 
Initializing the profile and prefetch data for this app 



5 



6. Performing any required custom installation tasks 



7 



Displaying the comment to the user if required 

8. Completing the installation 

9. Rebooting the computer if necessary 
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AIM'S policy is that if it encounters any fatal error during the execution of ALMInstal- 
lApphcation, it will attempt to undo everything it did before returning. AIM also grace 
folly handles aborted installs and uninstalls. 



Step 1 - Preparing for the installation 



First, AIM checks if the application is already installed by looking for an Appld registry 
key for the specified Appld. If found, then the AppInstalllnProgress registry value is 
checked. If it exists and is l , the user is asked if he wants to re-install, otherwise, he is 
asked to restart an aborted or damaged installation. If the user says no, AIMInstallAp- 
plication cleans up and exits with an error. 

Next, a free disk space check is performed to ensure that enough disk space is available 
for the install. The available free space must be at least twice the size of the Applnstall- 
Block to proceed. 

Next, an Appld folder is created for the app (described earlier), and the AppInstallBlock 
file is copied to this folder. AIM then opens the AppInstallBlock using AIMscOpenAp- 
plnstallBlock. Then the Appld registry key is created and the four defined values created 
and initialized. The AppInstallState value in particular is set to 1 to indicate an install is 
m progress. If any of these operations fail, AIMInstallApplication cleans up and exits 
with an error. 



Step 2 - Displaying the license 

AJMscEnforceLicenseAgreement is called to display the license text to the user and ask 
lor his agreement. If the functions fails or if the user's response is returned as FALSE 
AIMInstallApplication cleans up and exits with an error. 

Step 3 - Installing local files 

The install log file to be used for this application is created or open and truncated AIM- 
scInstallAppFiles is called to copy the install files to the computer and to create the 
spoof entries specified in the AppInstallBlock. Handles to the spoof subkey and the spoof 
refcount subkey are opened and passed to this function, as well as a path to the newly 
created mstall log file. If the function fails, AIMInstallApplication cleans up and exits 
with an error. 



If it succeeds, a boolean is returned indicating whether a reboot needs to occur due to 
shared files being overwritten. This value is remembered for use in step 10. 

Step 4 - Modifying the registry 

AIMscInstallApp Variables is called to perform the registry modifications specified in 
the AppInstallBlock. If the function fails, AIMInstallApplication cleans up and exits 
with an error. 
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Step 5 - Initializing profile/prefetch data 

AIMscInstallAppPrefetchFile is called to create and initialize the prefetch file for this 
application. The file has the structure specified in the Data Structures section of this 
document. This function takes a path to the prefetch file to be created. If the function 
fails, AIMInstallApplication cleans up and exits with an error. 

Step 6 - Performing custom install tasks 



AIMscCallCustomlnstall is called to extract the custom code .dll contained in the Ap- 
plnstallBlock and to call the InstallQ function it exports. If AIMscCallCustomlnstall 
fails, AIMInstallApplication cleans up and exits with an error. 

Step 7 - Displaying a comment 

AIM scDispl ay Comment is called to display any comment to the user contained in the 
appropriate section of the Applnstallblock. If this function fails, AIMInstallApplication 
cleans up and exits with an error. 



Step 8 - Completing the installation 



The AppInstalllnProgress registry value is set to 0 to indicate the install is complete. 
AIMscOpenAppInstallBlock is called to close the AIBFileRef opened in step 1, and any 
handles to open registry keys are also closed. 

Step 9 - Rebooting the computer (if necessary) 

If AIMscInstallAppFiles in step 3 returned a value indicating a user reboot is necessary, 
or if AIMscGetAIBShouldReboot is called and returns a value of TRUE, the user is 
asked to reboot. Otherwise, no reboot is performed and the application is ready to be run. 
AIMInstallApplication exits returning SUCCESS (0). 

AIMUninstallAppIication Prototype 

Uninstalling an eStream application consists of the following steps: 

1 . Preparing for the uninstallation 

2. Undoing all modifications done to the registry during install 

3. Undoing all file copies performed during install and removing spoof entries for 
this app 

4. Deleting the profile/prefetch data for this application 

5. Performing any required custom uninstallation tasks 

6. Completing the uninstallation 

7. Rebooting the computer if necessary 
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If the uninstallation fails for any reason, AIMUninstallApplication will tell the user that 
the umnstall has failed and that he should attempt to re-install the application before try- 
ing to uninstall again. 

Step 1 - Preparing for the uninstallation 

First, AIM checks if the application is already installed by looking for the Appld registry 
key corresponding to the specified Appld. If not found, then AIMUninstallApplication 
exits with an error. 

Then, the AppInstallState value is set to 2 to indicate an uninstall is in progress AIM- 
scOpenAppInstallBIock is called to open the AppInstallBlock at the path specified by 
the AppInstallBlockPath key. If this fails, then AIMUninstallApplication exits with an 
error. 

Step 2- Undoing registry modifications 

AIMscUninstallAppVariables is called to reverse the registry modifications specified in 
the AppInstallBlock. If the function fails, uninstall cannot proceed safely and AIMUnin- 
stallApplication exits with an error. 

Step 3 - Undoing file copies and removing spoof entries 

AIMscUninstallAppFiles is called to delete the files copied during install and to remove 
the spoof entries written then. Handles to the spoof subkey and spoof refcount subkey are 
passed to this function, and are where the spoof entries are removed from. If the function 
fails, uninstall cannot proceed safely and AIMUninstallApplication exits with an error. 

Step 4- Deleting profile/prefetch data 

AlMscUninstallAppPrefetchFile will be called to remove the prefetch data stored for 
this application. Any failure is ignored. 

Step 5 - Performing custom uninstall tasks 

AIMscCallCustomUninstall is called to extract the custom code .dll contained in the 
AppInstallBlock and call the UninstallQ function it exports. If AIMscCallCustomUnin- 
stall fails, uninstall cannot proceed safely and AIMUninstallApplication exits with an 



error. 



Step 6 - Completing the uninstallation 

AIMscGetAIBShouIdReboot is called and the return value saved. Then AIMscO- 
penAppInstallBlock is called to close the AIBFileRef opened in step 1, and the Appld 
folder and all its contents are deleted. The Appld registry key and all its subkeys are de- 
leted also. Any handles to open registry keys are closed. Any failures here are ignored 
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Step 7 - Rebooting the computer (if necessary) 

If AIMscUninstallAppFiles in step 3 returned a value indicating a user reboot is neces- 
sary, or if AIMscGetAIBShouldReboot is called and returns a value of TRUE, the user 
is asked to reboot. Otherwise, the uninstallation is complete. AlMUninstallApplication 

exits returning SUCCESS (0). 

AIMsc Function Prototypes 

Prototypes for the AIMsc functions declared earlier are given in this section. 
UINT32 

AIMscOpenAppInstalIBIock(LPCTSTR PathToAIB, AIBFileRef *pAIBFile) 

First, the file at PathToAIB is opened. Then, the header is read in, header version and size 
is verified, and section sizes and offsets are verified. An opaque pointer to an AD3- 
Filelnfo structure is returned in the pAIBFile parameter. 

UINT32 

AIMscCloseAppInstallBIock(AIBFileRef AIBFile) 

The file handle at ((AIBFilelnfo *) AJBFile)->Fi]eHandle, and the AIBFile structure is 
freed. 

void 

AIMscGetAIBVersion (AIBFileRef AIBFile, UINT32 *pAIBVersion) 
void 

AIMscGetAIBAppId(AIBFileRef AIBFile, UINT8 pAIBAppId[16J) 
void 

AIMscGetAIBVersionNo(AIBFileRef AIBFile, UINT32 *pAIBVersionNo) 
void 

AIMscGetAIBShouldReboot( 
AIBFileRef AIBFile, 
BOOLEAN *pAIBShouldReboot) 

UINT32 

AIMscGetAIBAppName( 
AIBFileRef AIBFile, 
LPTSTR pAIBAppName, 
UINT16 *pSizeAJBAppName) 
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These four functions are trivial. They directly return the corresponding value of the vari- 
able in ((AEBFilelnfo *) AIBFile)->AIBFileHeader. (See the interface declaration for 
AIMscGetAJBAppName for details on its calling logic.) 

UINT32 

AIMscCheckAIBCompatibleOS( 
AIBFileRef AIBFile, 
BOOLEAN *pWasOSCompatible) 

This function will call an API such as GetVersionEx (for Windows) to determine the cur- 
rently running operating system. The OS version is then converted to a bitmask (using 
constants defined in an external AppInstallBlock header file) and compared with the OS 
and Service Pack bitmaps in ((ABFilelnfo *) AIBFile)->AIBFileHeader. If the bits are 
present, pWasOSCompatible is set to TRUE, otherwise FALSE. 

UINT32 

AIMscInstallAppFiles( 

AIBFileRef AIBFile, 

HKEY SpoofKey, 

HKEY SpooIRefCountKey, 

LPCTSTR InstallLogFile, 

BOOLEAN *pIsRebootNeeded) 

The index entry array at ((ABFilelnfo *) AIBFile)->IndexEntries is scanned to find the 
File section. If not found, an error code is returned. Otherwise, the section is parsed. 

The File section is organized as a series of trees, with directories as non-leaf nodes and 
plain .files as leaf nodes. All nodes are stored contiguously according to the pre-order tra- 
versal of the trees. 



Each directory node contains the name of a single directory and a number indicating the 
number of children this node has. Each file node contains the file version and file name, 
and a flag indicating whether the file is to be spoofed or not. If so, then the last entry in 
the node is the spoofed pathname, otherwise it is the actual contents of the file itself. 

(The actual structure types defined for these nodes are assumed to be defined in a header 
file external to AIM. See the AppInstallBlock-LLD for reference.) 

A directory stack algorithm will be used to parse the trees and reconstruct the directory 
paths. Due to the complexity of the task, several helper functions are used by the algo- 
rithm to partition this work. 

For every file copied or spoof entry added by the algorithm, an entry is made to the file at 
InstallLogFile. For the sake of brevity, no mention is made of the logging in the pseu- 
docode below. 
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The parsing algorithm is as follows (TOS refers to the node at the top of the stack): 

empty out directory stack 
while there are nodes to read in the File section 
read a node 

if the node is a directory 

HandleDirectoryNode(node, . . .) 
else 

HandleFileNode(node, . . .) 

while stack is non-empty and TOS node number of children is 0 
pop directory stack 
if stack is non-empty 

decrement number of children in TOS node 

Here is HandleDirectoryNode(node): 

if node directory name contains Builder/AIM defined variables 
replace variable substrings with local expansions 

if directory stack is empty 

if node directory name is not fully qualified 
error 

push onto directory stack an entry with: 

- node directory name 

- node number of children 

else 

push onto directory stack an entry with: 

- "TOS directory name" cat "directory name" 

- node number of children 

Here's HandleFileNode(node, . . .): 

if node filename contains Builder/AIM defined variables 
replace variable substrings with local expansions 
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if directory stack is empty 

if node filename is not fully qualified 
error 

call DoFileInstall(filename, node, ...) 

else 

if number of children in TOS node is <= 0 
error 

call DoFileInstall('TOS directory name" cat "filename", node, ...) 
decrement the number of children in TOS node 

Here's how DoFileInstall(filename, node, . . .) works: 

if the file node is a spoof entry 
if filename already exists 
if existing version is earlier 
mark for spoofing 
else // filename does not exist 

create zero-length file at filename 
mark for spoofing 

else // file will be copied not spoofed 
if filename already exists 
if this file is a .dll 

increment .dll shared ref count in registry 
if existing version cannot be read or existing version is earlier 
mark for copy 
else // filename does not exist 

add line to FilesLogPath file containing filename 
mark for copy 

if marked for spoofing 

create spoof entry under SpoofKey 

create or update spoof refcount under SpoofRefCountKey 

if marked for copy 

attempt to copy node file to client computer 
if copy fails 
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tell system to perform copy at reboot 



The pIsRebootNeeded argument will be set to TRUE if any file copies were scheduled to 
happen at reboot, FALSE otherwise. Additionally, if any spoof entries were added, then 
an IOCTL will be sent to the spoof driver asking it to reload the spoof database. 

The shared .dll reference count mentioned in the algorithm above is stored in a standard 
place in the Windows registry. AIM will create or increment this reference count for 
every non-spoofed .dll included in the AppInstallBlock (they can all be potentially shared 
since they will be placed outside of the eStream app directory). Each such .dll has an as- 
sociated REG DWORD value under the key at: 

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\ 
SharedDLLs 

The value's name is the path to the .dll and the value's data is a integer that is the refer- 
ence count for this .dll. 



UINT32 

AJMscUninstalIAppFiles( 

AIBFileRef AIBFile, 

HKEY SpoolKey, 

HKEY SpoofRefCountKey, 

LPCTSTR InstallLogFile, 

BOOLEAN *pIsRebootNeeded) 

Currently, the AIBFile parameter is not even needed since all of the information needed 
for file unmstall is contained in the log file at InstallLogFile. However, it is included to 
enforce a design decision, which is that the user should have a valid reference to an Ap- 
pInstallBlock before performing any install/uninstall related actions on an eStream app. 

The algorithm for AIMscUninstallAppFiles is simple. It iterates over the change entries 
contained m the log file, and undoes file copies and spoof entry additions when it is safe 
to do so. Here is the algorithm: 

while there are change entries in the log file 
read the next entry 



if the entry is of the form "ADDED <filename>" 
if filename is a .dll 

decrease refcount of .dll 
if refcount is 0 

mark for deletion 
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else // file not a .dll 
mark for deletion 

if file is marked for deletion 
attempt to delete file 
if deletion fails 

tell system to perform deletion at reboot 

else if the entry is of the form "OVERWROTE <filename>" 
if filename is a .dll 

decrease refcount of .dll 

else if the entry is of the form "SPOOFED <filename>" 
decrease refcount of spoof entry for this filename 
if refcount is 0 

delete the spoof entry 

if O-byte placeholder at filename still exists 
delete it 



A failure to delete a file or to schedule its deletion will not cause AIMscUninstal- 
lAppFiles to fail. 

UINT32 

AIMscInstallAppVariables( 
AIBFileRefAIBFile, 
LPCTSTR InstallLogFile) 

The index entry array at ((AIBFilelnfo *) AIBFi]e)->IndexEntries is scanned to find the 
Variable section. If not found, an error code is returned. Otherwise, the section is parsed. 

The Variable section is organized as a series of trees, similar to how the File section is 
organized. There are two types of nodes, key nodes and value nodes. Registry keys can 
contain other keys and registry values, while registry values are just name/data pairs that 
are stored in keys. A registry value name is always a string, but its data can be stored as 
any one of a number of types. 

Non-leaf nodes must be key nodes, while leaf nodes can either by key or value nodes All 
nodes are stored contiguously according to the pre-order traversal of the trees. 
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(The actual structure types defined for these nodes are assumed to be contained in a 
header file external to AIM. See the AppInstallBlock-LLD for reference.) 

A keyhandle algorithm will be used to parse the trees and create the registry keys and 
values. Due to the complexity of the task, several helper functions are used by the algo- 
rithm to partition this work. 

For every key or value added by the algorithm, an entry is made to the file at InstallLog- 
File. For the sake of brevity, no mention is made of the logging in the pseudocode below. 

The parsing algorithm is as follows (TOS refers to the node at the top of the stack): 

empty out the key handle stack 
while there are nodes to read in the Variable section 
read a node 

if the node is a key 

HandleKeyNode(node, . . .) 
else 

HandleValueNode(node, ...) 

while the stack is non-empty and the TOS number of children is 0 
if TOS key handle is open 

close it 
pop the directory stack 
if the stack is non-empty 

decrement the number of children in TOS 

Here is HandleKeyNode(node): 

if the keyname contains a Builder/AIM defined variable 
replace the variable substring with its local expansion 

if the keyhandle stack is empty 

if the key name is not fully qualified 
error 

create key under HKCR, HKLM, etc. and save key handle 

else 

if the number of children in TOS is <= 0 
error 
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create key under TOS key handle and save key handle 

push onto the keyname stack an entry with: 

- open key handle 

- this number of children 

Here's HandleValueNode(node, . . .): 

if the keyname stack is empty 
error 

else 

if the number of children in TOS is <= 0 
error 

call Dolnstall Value(TOS key handle, node, . . .) 
decrement the number of children in TOS 

Here's how DoInstallValue(key handle, value node, . . .) works: 

if the value name contains a Builder/AIM defined variable 
replace the variable substring with its local expansion 
call SetValueEx(key handle, 'Value name", value type, value data, . 

UINT32 

AIMscUninstallAppVariables( 
AIBFileRef AIBFile, 
LPCTSTR InstallLogFile) 

Currently, the AIBFile parameter is not even needed since all of the information needed 
for file uninstall is contained in the log file at InstallLogFile. However, it is included to 
enforce a design decision, which is that the user should have a valid reference to an Ap- 
plnstallBlock before performing any install/uninstall related actions on an eStream app. 

The algorithm for AIMscUninstallAppVariables is simple. It iterates over the change 
entries contained in the log file, and undoes registry key and value additions when it is 
safe to do so. Here is the algorithm: 

while there are change entries in the log file 
read the next entry 

if the entry is of the form "ADDED <keyname>" 
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if key at keyname (fully qualified) still exists 
delete it and all subkeys and values 

else if the entry is of the form "ADDED <valuename>" 
if value at valuename (fully qualified) still exists 
delete it 



Keys and values that were overwritten are not deleted, which is why those log file 
entries are not considered by the algorithm above. 

A failure to delete one or more registry entries will not cause AIMscUninstallAppFiles 
to fail. 

UINT32 

AIMscInstallAppPrefetchFilefAIBFileRef AlBFile, LPCTSTR PrefetchFile) 

The index entry array at ((AlBFilelnfo *) ABFile)->IndexEntries is scanned to find the 
Prefetch section. If one is found, the prefetch data is read in and written out into the file at 
PrefetchFile as an array of Prefetchltem structures (this structure will change to match 
how the prefetch data items are represented in the AppInstallBlock-LLD). Any existing 
file at PrefetchFile is overwritten. 

Next, the Prefetch component is called to set up an association between the new applica- 
tion and its prefetch file. 

UINT32 

AJMscUninstalIAppPrefetchFHe(AIBFiIeRef AlBFile, LPCTSTR PrefetchFile) 

The file at PrefetchFile is deleted and the Prefetch component is called to remove the as- 
sociation between the app being uninstalled and the prefetch file. 

UINT32 

AIMscInstallAppProfileFilefAIBFileRef AlBFile, LPCTSTR ProfileFile) 
UINT32 

AIMscUninstalIAppProfileFiIe(AIBFileRef AlBFile, LPCTSTR ProfileFile) 

(NOT FUNCTIONAL IN ESTREAM 1.0) 

UINT32 

AIMscCaIlCustomInstall(AIBFileRef AlBFile) 

The index entry array at ((AlBFilelnfo *) AIBFile)->IndexEntries is scanned to find the 
Code section. If one is found, the section is read in and written out again as a .dll library. 
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This library is loaded and the InstallQ function export is called (and its return value re- 
turned). 

UINT32 

AJMscCallCustomUninstall(AJBFileRef AIBFile) 

The index entry array at ((AIBFilelnfo *) AIBFile)->lndexEntries is scanned to find the 
Code section. If one is found, the section is read in and written out again as a .dll library. 
This library is loaded and the UninstallQ function export is called (and its return value 
returned). 

UINT32 

AIMscEnforceLicenseAgreement( 
AIBFileRef AIBFile, 
BOOLEAN *pBUserAgreed) 

The index entry array at ((AIBFilelnfo *) AffiFile>>IndexEntries is scanned to find the 
LicenseAgreement section. If one is found, the license text is read in and displayed to the 
user in a dialog. The user will be asked to either agree or disagree with the license, and 
pBUserAgreed will reflect his decision. 

UINT32 

AIMscDisplayComment(AIBFileRef AIBFile) 

The index entry array at ((AIBFilelnfo *) AL3File)->IndexEntries is scanned to find the 
Comment section. If one is found, the comment text is read in and displayed to the user in 
a dialog. 

Testing design 
Unit testing plans 

AIM will be tested by a program that generates AppInstallBlocks designed to stress the 
component. AIM will be asked to install the given AIB and if successful, the resulting 
state of the system will be compared to the expected state had all the files and variables 
been installed correctly. An uninstall will then be performed and the system state also 
checked. 

The focus of the testing will obviously be on the File and Variable sections. The other 
sections such as Code and Comments will be stressed also, but their boundary conditions 
are much simpler. 

AIM's ability to gracefully handle aborted installs and uninstalls will also be tested. 
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Stress testing plans 

The program described above can be deliberately tuned to create AppInstallBlocks of un- 
usual size and organization. For example, AppInstallBlocks with thousands of files and 
registry entries, or files and entries with unusually long names, etc. 

Coverage testing plans 

In addition to the stress testing, deliberately malformed AppInstallBlocks will be gener- 
ated by the test program to hit as much error-handling code as possible. AIM's data files 
and registry entries can also be deliberately mangled to help achieve this effect. 

Cross-component testing plans 

As soon as they are available, Builder-generated AppInstallBlocks will be tested to verify 
that the AIM is compatible with the Builder's output. As soon as the LSM and a browser 
plugin are available, the communication path from browser to LSM to AIM will be 
tested. As soon as the file spoofer is available, compatibility with the file spoof entries 
that AIM makes will be tested. 

Upgrading/Supportability/Deployment design 

AIM will make use of the eStream logging facility to record information about errors and 
other unusual conditions that occur. The log file will be useful for diagnosing problems 
that occur during testing and in real world situations. 

If the AIM component is upgraded, it must still be able to uninstall any eStream applica- 
tions installed at the time of upgrade. This entails being able to interpret old AIM registry 
entries and data files, including the AppInstallBlock. This is more a concern for future 
designers of the AIM component, however. 

Open Issues 

o How will the various anti-piracy strategies being considered affect 
the design of AIM, if at all? 
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Client Senarios - Install, Upgrade and Configure 

Version 1 .0 



Note that these sequences do not include the various places at which the installer should 
stop and ask for input from the user. It is intended to describe the installation and 
upgrade process from a technical standpoint. The UI must be described elsewhere. 

Note that not all decision points are described. I ignore ones that don't have a material 
impact on the scenarios at hand. For example, when I say that the user downloads the 
eStream client installation program from the ASP's web site, this does not preclude the 
installer being delivered on other media, such as via ftp or physical media. It also 
includes the case of the system administrator acting on behalf of the user. When I say the 
client installer is unpacked to the user's disk, this covers any mechanism by which the 
user might run the installer, such as from a network share. 

The packaging of the eStream client software installer is platform-dependent. On 
Windows, the installer will most likely be distributed as a .ZIP file or a self-extracting 
executable. On Unix platforms, the installer will probably be a .RPM or a gzipped tar 
archive. 

Scenario 1- Install eStream client SW, no previous installation 

This scenario is for the normal installation process. The machine does not have eStream 
installed on it. Client installation should only be performed once per machine (assuming 
that the client is not uninstalled.) 

0. (Not shown on diagram.) ASP makes eStream client software available on its web 
site. The client SW install package may be generic, or it may be pre-customized for the 
ASP with settings for the ASP f s servers. 

1. The user downloads the eStream client install application from an ASP via a web 
browser. 

2. The installer is copied to the user's hard drive and extracted. (Optionally, the installer 
can be accessed directly from a network share, as is the likely case in a corporate 
intranet.) 

3. The installer is run the by the user (or by the system administrator on behalf of the 
user.) The installer queries the registry to determine if the eStream client has already 
been installed on this machine. Since this is a new installation, it will find that the 
registry keys for the eStream client are undefined. If there are registry settings for other 
installed software that may influence what the eStream installer will do, those are 
checked at this step as well. Note that the installer may choose different actions or to 
install different things depending on the version of the operating system that is present. 



4. Just to be sure, the installer checks to see if any of the eStream client drivers are 
already installed on this system. (Note that we thus need some mechanism for 
determining which of our drivers is installed on a client system, and the versions of these 
drivers. Two possibilities are to look for the files on disk or to attempt to access the 
loaded drivers.) Since this is a new installation, none of the drivers will be on the system. 

5. The installer copies the eStream drivers to the appropriate places on the user's system 
so that they can be loaded on the next system reboot. (If the drivers can be installed 
without rebooting the system, this is preferred.) 

6. Client user-mode components are installed on the client system, in a user-specified 
location. The browser plugin (if we provide one) is installed for the user's default 
browser, or optionally, for any other supported browser. 

7 System files (the registry and possibly other files) are modified so that the drivers will 
be loaded on the next system reboot. System files are modified so that the user-level 
components will be started on boot or logon, if that has been requested. Default 
configuration information (either specified by the user, or as customized by the ASP) will 
be written into the appropriate eStream configuration files, or into the registry. Uninstall 
information is written into the appropriate place. 

8. Initial (empty) versions of the cache, application registry information, and application 
spoof information are created or installed. Alternatively, the client software could know 
how to create these files if they are not found when the software is started. 

9. (Not shown on diagram.) The drivers are loaded into the running system, or the 
system is rebooted. 

Needed APIs 

None specifically needed. 

Scenario 2 - Upgrade eStream client SW (trivial case - same or newer 
version installed) 

This scenario covers the case when the user attempts an install or an upgrade of the 
eStream client components, when the version already installed is at least as new as the 
version they are attempting to install. This may occur if a user does not know that 
eStream is already installed on a machine, or if they download and run the installer to 
upgrade to the latest version, when they are already running the latest version. The 
installer or upgrader should determine that there is no point in doing an upgrade and 
gracefully exit. Note that Scenario 5 covers the case where the user elects to force a 
reinstall or a downgrade of the client software. 

0. (Not shown.) ASP provides an eStream client software install or upgrade program via 
their web site. 



1 . User downloads the eStream client software from the ASP web server. The 
installation or upgrade program is for a version no newer than the eStream software 
already installed on the client system. 

2. The install or upgrade SW is extracted to the client machine's hard disk. 

3. The install or upgrade software checks the registry to see if there is an installed 
version of the client at least as new as the installer. There is, so the installer notifies the 
user that a newer version is already installed. The user elects not to reinstall the client 
software, and the installer exits. 

Needed APIs 

The installer or upgrader must determine the version of the client software currently 
installed. Most likely, this will be done through the registry, so no APIs are specifically 
needed. 

Scenario 3 - Upgrade eStream client SW (easy case - no kernel 
components needed) 

This scenario covers the case where the user installs an update to the eStream client 
software, but only the user-mode components are newer than those installed on the client 
system. In this case, it should be (theoretically) possible to replace only the user-mode 
components, and restart eStream without rebooting. This scenario is a special case of 
Scenario 4. We probably want to eliminate this scenario and always install fresh copies 
of all client system components (other than configuration related files). 

0. (Not shown.) ASP provides an eStream client software install or upgrade program via 
their web site. 

1. User downloads the eStream client software from the ASP web server. EStream is 
already installed on the client's machine. 

2. The install or upgrade SW is extracted to the client machine's hard disk. 

3. The install or upgrade software checks the registry, and finds that the installed client 
software is older than that provided by the installer. It also discovers that only user-mode 
components need to be replaced. 

4. The installer checks to see if the user-mode client components are running, and if so, 
if any bits are currently being served through them. If so, it brings up a dialog box asking 
the user to shut down any applications that may currently be accessing the z: drive. The 
upgrade may be cancelled at this point. Kernel components are notified that user-mode 
components will be coming down for an upgrade. All non-kernel client components are 
shut down. 



5. Client user-mode components are replaced, as necessary. 

6. The registry is updated with information about the newly installed client components. 
Any necessary changes to the uninstall information are made. 

7. Any persistent data (things in the cache, configuration files, etc.) that have changed 
format are converted to the new format, or discarded. (Alternatively, the client software 
could understand both the old and the new data format, and perform the conversion itself 
the next time it is run.) 

8. (Optional). User-mode components are restarted. No reboot is necessary. 
Needed APIs 

The Cache Manager must support a stop client API: 
bool StopClient() 

StopClient would return TRUE if the client has been stopped, and FALSE otherwise 
(perhaps because the user is currently running an eStreamed app, and doesn't want to stop 
right now.) 

Scenario 4 - Upgrade eStream client SW (most general case - kernel and 
user components) 

This scenario covers the case when kernel-mode as well as user-mode components must 
be replaced. I didn't create a separate scenario for the case where kernel-mode 
components must be replaced, but user-mode components don't need to be, since that 
case is really no simpler than this one. 

0. (Not shown.) ASP provides an eStream client software install or upgrade program via 
their web site. 

1 . User downloads the eStream client software from the ASP web server. 

2. The install or upgrade SW is extracted to the client machine's hard disk. 

3. The install or upgrade software checks the registry, and finds that the installed client 
software is older than that provided by the installer. It discovers that at least one kernel- 
mode component must be replaced. 

4. The installer checks to see if the user-mode client components are running. If 
necessary, it brings up a dialog box asking the user to shut down any applications that 
may currently be accessing the z: drive. The upgrade may be cancelled at this point. 



Kernel-mode components are notified that eStream is being taken down for an upgrade. 
All non-kernel client components are shut down, and drivers are unloaded, if possible. 

5. Client user-mode components are replaced, as necessary. Kernel-mode components 
are also replaced. It may be necessary that new versions of the drivers are placed in a 
special location, to be installed on the next reboot. 

6. The registry and other configuration files are updated with information about the 
newly installed client components. Any necessary changes to the uninstall information 
are made. 

7. Any persistent data (things in the cache, configuration files, etc.) that have changed 
format are converted to the new format, or discarded. (Alternatively, the client software 
could understand both the old and the new data formats, and perform the conversion 
itself.) 

8. (Not shown). Machine is rebooted, either immediately or some time later. On reboot, 
the new kernel-mode drivers are installed in the appropriate locations and loaded. 

Needed APIs 

See StopClientO above. 

Scenario 5 - Forced Reinstall/Downgrade of client SW 

Though we would normally prefer a user to uninstall eStream before installing an older 
version, forced reinstalls (and possibly downgrades) may be necessary for a variety of 
reasons. The most important situation in which we would want to support this is when 
the uninstaller fails for some reason, leaving the system in a partially-installed state. 
Note that users often need to reinstall software because configuration files or registry 
settings have become corrupted, leading to aberrant application behavior. Note that 
downgrades are not strictly safe, because newer versions may have made incompatible 
changes to persistent file formats. We should thus support a reinstall mode that replaces 
settings files with fresh, new ones. 

0. (Not shown.) The ASP provides an eStream client installer on their web site. 

1 . User downloads the eStream client SW installer. 

2. Installer is unpacked onto the client system's hard drive. 

3. The installer queries the registry and determines that it is not newer than the version of 
the client software already installed. The user specifically requests a reinstall or 
downgrade. 



4. The installer checks to see if the cache manager is running. If it is, it asks it to shut 
itself and all other user-mode components down. If necessary, the user will be prompted 
to shut down any applications currently accessing the z: drive. 

5. If the cache manager was running, it notifies the kernel components that the user- 
mode components are shutting down for replacement. 

6. (Not shown on diagram.) User components exit. 

7. User-mode components are all reinstalled, possibly with an older version. 

8. Kernel-mode components are all reinstalled, possibly with an older version. 

9. Application setup information is overwritten with default values, including empty 
versions of persistent caches, app install information, and app spoof information. 
Optionally, the user could request that the configuration information and cache not be 
overwritten. 

10. (Not shown.) The machine is rebooted immediately or later, and the new drivers are 
installed upon reboot. 

Needed APIs 

See StopClientO above. 

EStream Client SW Configuration 

We have not fully defined what aspects of the client software will be configurable. 
Certainly, cache size and location should be configurable. 

0. (Not shown.) The user brings up the eStream configuration UI via some mechanism. 

1 . The UI determines the current configuration and settings by querying the cache 
manager. If necessary, the UI starts the cache manager. 

2. The cache manager queries the running user-mode components and kernel-mode 
components to determine the current state of the system. 

3. Some of the queries may end up referencing on-disk configuration information 
through the client file manager. 

4. If the user changes a setting, the change is sent to the cache manager to take effect 
immediately. 

5. Notices are sent to user and kernel mode components to take effect immediately. 



6. Persistent changes are written to configuration files or the registry via the client file 
manager. 

Needed APIs 

We need APIs for the client UI to get and set program settings, and APIs to read and 
write these settings to persistent configuration files. The exact format of this APIs is a bit 
fiizzy now, since we don't know all of the things that we want to configure. 
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Introduction 

The following are scenarios or tasks we've identified that will take place on a client machine. Some of the following 
scenarios are explicitly requested by a user; others take place in the background without a user's knowledge. 

Some working definitions and assumptions here: 

• The eStream file system will be referred to as the Z: drive 

• A "user" is registered with an ASP. An "account" is a "billable unit" with an ASP. 

• For a given ASP, there's a many-to-many relationship between users and accounts (single user is a member of 
multiple accounts; an account is used by multiple users). 

• A "user identifier" is cached info on a client machine for a user. It identifies 

o user, password 

o ASP 

o account server 
o DRM server 
nstall/upgrade eStream client SW 

Tie first time eStream is installed, this is an explicit action by the user or a sysadmin. For upgrades, this could be 
xplicit, or the eStream client could detect that it needs to upgrade itself (possibly by asking permission first) Some 
ubcategones: 7 * 

• Install is requested, latest version of eStream already installed 

• User doesn't have sufficient permission to install 
Configuration of eStream client SW 

ither during installation, or subsequent to this, some client parameters need access. 

• Size, location of on-disk cache 

• Startup of eStream client: auto vs. manual 
tart eStream client SW 

his allows the client to query the account server for new subscriptions, to authenticate and start eStream'ed apps to start 
iy spoofers on the system, etc. FF ' 

le client can start either explicitly by the user, or implicitly at boot or login. 
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Manage "user" and "account" data 

Some subcategories: ^ 

Remove user data from an ASP 

Add/remove account that this user is a member of 

Subscribe/unsubscribe to an application 

This can come from "within" eStream or not. I.e., the client SW may not be informed that a new subscription has been 
made (e.g., it s been done on the ASP web site); hence the subscribed app cannot be immediately installed. 

Account/user queries 

• billing info 

• subscription info 

"Install" account information on the client 

rhis means to cache the "user identifier" on the client machine. This allows the eStream client do to asynchronous 
queries for new subscriptions, transparently start subscribed apps, etc. 

installing account info is inherently an explicitly requested action. We identified two possible scenarios: 

• using a dialog to identify and "log in" to an ASP account server 

• export an existing "user identifier" to a file that could be transferred to another machine, and used by the eStream 
client to install the same identifier on this second machine (e.g., by double-clicking on it) 

'Install" a subscribed application 

This means: download all necessary bits to make a subscribed app "ready-to-run. " 

liis can be explicit (e.g., when a user subscribes to a new app) or implicit (e.g., the client SW pings the account server to 
Dok for a new subscription). 

Juery Z:\ (i.e., the root directory of the eStream FS) 

Ve shouldn't require authentification if only the top-level components of the root director/ are queried. Ideally the client 
W simply identifies the contents with some representation of the actual directories - e.g., with a special icon displayed 
1 Explorer. . * J 

)uery Z:\.* (i.e., a file/directory under the root directory) 

his includes actually running applications! 
.ccessing actual application directory contents: 

• May need to authenticate before granting access 

o authentication may fail 

• May not need to authenticate; e.g., 

o already granted within time slice 
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o already using files within the application hierarchy 

Aim Explorer at a folder with a file that's associated with an eStream'ed app 
Termination of an application 

• Is this the last open file associated with an application hierarchy? 

• Do we need to upload user data? 

• Do we need to contact DRM server to give up authentication token? 
Uninstalling components 

Uninstalling a subscribed app 

Unclear how clean we need to leave the system 

Uninstalling eStream client components 

Unclear how clean we need to leave the system 

Failures/errors 

App crashes 

eStream crashes 

Client machine crashes 

Kill a zombie connection 

Unexpected loss of connectivity 
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Introduction 



Ihis is an initial requirements/design document for the eStream file system driver (EFSD), implementing the eStream file 
system (EFS), for a Windows 2000 client machine. 

Very high-level view 

On the client side of the eStream product, the user will have a networked drive visible and accessible; this drive will 
'contain" all applications available in the current user's session. There will be limitations on what will be visible to a user 
within this drive, and there are uncertainties as to how the entire client design will lay out. However, it's still useful to 
specify some known issues and suggestions for how the EFSD should be designed. 

Here's the general overview of what will reside on the client and what the control flow will be. 

• The eStream drive will be managed by the eStream FSD, which is a form of network redirector. It will interface 
with the I/O manager, Cache manager, and VM manager. 

• Requests for file and directory contents will be passed from the EFSD to what I'm calling the "eStream Client 
Manager," or ECM, which may reside in either user or kernel space. 

• The ECM will handle, at least: 

o passing necessary read/write data and metadata requests to the server 

o caching all pertinent data and metadata 

o performing any profiling and prefetch work 

o helping the eStream FSD keep coherence with the server 

• The network interface between the ECM and the server is the subject of other design documents 

Although in practice it may be that much of the eStream drive will be read-only from a user's perspective, the EFSD must 
)e designed for full write capabilities. Policies for read/write access will be the responsibility of the ECM and the server. 

Design overview 

interfacing with the client OS 

rhe W2000 EFSD will handle all appropriate requests from the various Windows Executive components. Here is a list of 
equests it needs to handle: 

• Create IRPs, for both new and existing files 

• Cleanup, Close IRPs 

• Read and Write IRPs: 

o synchronous and asynchronous 
o cached and non-cached 
o paging and non-paging 

• Fast I/O reads and writes (with buffers or MDLs) 
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• File information (get and set) IRPs 

• Directory query IRPs 

• Volume information (get and set) IRPs 

• File system information (get and set) IRPs £ % 

• Various Device control IRPs (as needed by our implementation) 

• Flush buffer IRPs 

• System shutdown IRPs 

• Various Fast I/O queries 

In particular, I expect we do not need to handle Directory Notification IRPs, though this is potentially an open issue. 

I'm also proposing we not support hard links on a Windows client (these are supported natively on NTFS on W2000 
only). 

ITie question of Byte-lock IRPs is an open one. These would probably be expensive to properly support, but it's unclear a 
this time if not supporting them will be a showstopper for us or not. 

Interfacing with the ECM 

How the eStream FSD will interface with the ECM will depend on whether the ECM is a user or kernel space service. 
Mevertheless, the data that needs to be exchanged between the two components can be described today. 

The EFSD and the ECM need to communicate all the basic information above, including: 

• Create requests 

• Open requests 

• Read/write requests 

• Directory content requests 
Rename requests 

• Delete requests 
File/directory metadata requests 

• Buffer flush requests 

• Volume allocation information requests (possibly) 

n addition, the ECM must be able to inform the EFSD that a file or directory it has open is either: 



not being modified by another process on any system; or 
potentially being modified by another process on another system 



Tiis is necessary for the EFSD to allow caching of the file on the client machine. 
Vhat's cached where? 

)ue to the requirements of the NT executive components, the EFSD needs to cache various items of file metadata in its 
wn data structures. As long as it's tracking some attributes, it makes sense to keep track of all that it easily can. In 
articular, the eStream FSD will keep track of: 



Time stamps: creation, last write, last access, last change other than write 

File sizes: allocated, actual (aka EOF) 

File attributes: e.g., normal, directory, read-only, hidden 



he EFSD will not cache directory contents information; this will be the task of the ECM. Given how frequent these 
^quests come through to the file system on Windows, we will definitely need to cache this on the client side. 
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In no particular order, here are some items that must be implemented in the eStream FSD. 

1 . There are no volumes, and no VPB for a network redirector; I've verified this with the LanManager redirector. 

a. We do need to understand how we can have the EFSD handle multiple mounted drives if needed, and how to 
distinguish them by name in the Create IRP. 

b. We don't have to support any operations on a volume in EFS. 

2. We should disallow the creation of paging files in EFS. There is a bit available for a Create IRP that specifies this, 
and we can complete the IRP with an unimplemented return code. 

3. All file synchronization will be on an FCB basis, using the standard Resource and PagingloResource ERESOURCE 
objects used by the rest of the Windows Executive. 

a. User requests will be synchronized by acquiring the main Resource - shared for reads, exclusive for writes, 
other changes, deletion, etc. 

b. Paging I/O requests will be synchronized by acquiring the PagingloResource - shared for reads, exclusive foi 
writes. Also exclusive access will be needed to set file sizes. 

4. Most disk file systems have a resource associated with a VCB, which is acquired exclusively for creation/deletion 
etc. We will have a global EFS resource for this, since there are no VCBs. 

5. Asynchronous requests will be handled by posting the IRP to the CriticalWorkQueue, and marking the IRP as 
pending. 

a. A common worker routine will be used for all async posts, which will dispatch the IRP to the appropriate real 
IRP routine when it's invoked. 

b. An async request will be defined as one that IoIsOperationSynchronousO returns false, and the EFSD is the 
top-level component (see below) 

6. The EFSD will track the top-level IRP for the thread whose context it is running in. In particular, 

a. No async processing request will be honored unless the EFSD is the top-level component 

b. No cache manager requests will be made unless the EFSD is the top-level component 

c. The top-level component is expected to acquire the appropriate resources, and the EFSD must not try to 
acquire them as well if it is not top-level 

d. EOF file size will not be extended or changed by paging I/O 

7. EFS will not support holes in files, and hence the ValidDataLength FCB field will be set to disable this. 

8. Most fast I/O routines will be supported in EFS. We should be able to use the FSRTL supplied routines for fast 
reads and writes. 

9. AH cache manager resource acquire/release callbacks will be supported. All will point to common routines that 
simply acquire or release the main Resource for the FCB. The Context pointer passed into all of them will be the 
FCB for the stream. 

10. Synchronous read/write requests will update the CurrentByteOffset in the File object 

1 1 . Each Create will result in a unique CCB data structure; this will be small, and only hold those few fields needed: 

a. For the Directory Control IRP, a CCB needs to hold the current entry index and the pattern originally used - 
for subsequent queries 

b. A field for various flags 

12. A single FCB will represent all current open stream instances of a file. When a new file is opened, the EFSD will 
search the current open FCBs to find one matching this file/directory name. 

a. For now, this will be a simple linked list with a linear search. We can improve this as needed 

b. The EFS global resource must be acquired exclusively: 

i. before the global FCB "list" is searched 

ii. before a new FCB is added to the list 
iii. before an FCB is deleted from the list 

13. EFS will not support open by file ID; hence the Filelnternallnformation class for a File Information IRP will not be 
supported. 

14. Actual I/O will be directed to standard routines in a separate file, so they can be isolated and updated easily as our 
method of transferring data changes. 
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1 5. Here's how to do file/directory renames: 

a. The I/O manager will send to the EFSD this sequence: 

i. Create for source 

ii, Create for target, with the SLOPENTARGETDIRECTORY flag set 

hi. Set Information with a Rename request for the source, sending the target directory FileObject handle, 
and the target filename in the file info record. 

b. EFSD needs to do this: 

i. When it receives the Create for the target and the target directory exists, return STATUS SUCCESS, 
and change the name in the FileObject to the basename of the target (the full pathname of the target is 
sent in), and set the Status.Information to FILE EXISTS or FILE DOES NOT EXIST, as appropriate. 
If the target directory doesn't even exist, return PATH NOT FOUND. 

ii. When it receives the Set Info request, if all the flags check out (e.g., if the file exists, ReplaceExisting 
must be TRUE), send a Rename request to the ECM. 

16. Reads and writes to only regular files will be supported, not to directories. 

17. Any code that touches user buffers or can call routines that may throw exceptions must be guarded by a try/except 
block. 

18. Some tips on memory allocation (from osrdocs/defensive-driv.html ) 

a. Use our own memory allocation/deallocation routines, instead of ExAllocatePoolQ et al. directly 

b. These routines can do various checks for trashing memory: 

i. fill allocated memory with a pre-defined bit pattern, instead of zeroes; fill deallocated memory with a 
different pattern. 

ii. allocate a header/trailer with standard information, like where allocated, from what pool, etc. 

iii. change the bit pattern in the header/trailer on deallocation, and look for freeing memory twice 

Design outline 

The EFSD will look a lot like the sample FSD from Rajeev Nagar's NT FS book, which looks a whole lot like the 
FASTFAT FSD source from the IFS kit. 

Most IRPs will have essentially two routines each to handle them: a dispatch routine which is sent the IRP directly, and a 
eal routine that does the actual processing. The dispatch routine is essentially a stub that calls the real one. The real one 
s used to handle async processing of the IRP by a worker thread. 

OriverEntry 

This does a whole slew of initialization, including the dispatch table, fast I/O table, the cache callbacks, the FCB list and 
ts synchronization object, creates the FS device object, and sets up the interface with the ECM. 

Create 

There is one Create routine; there will be no async processing of Create requests. Ultimately, its job is to send a create or 
>pen request to the ECM, and return SUCCESS or not to its caller. It also does at least the following: 

• The global FCB data structure synchronization object must be acquired exclusively 

• Paging files are not allowed 

• Write-through requests result in no caching 

• Related file objects must be for a directory 

• An absolute pathname will be generated if a related file object is specified 

• It will request to open/create the file from the ECM, and if successful, get the attributes it needs for the FCB 

• The FCB must be searched for by name in the global data structure; if found, this is used, else a new one created. 

• A new CCB must be created for this file 

• If open existing entry is specified, and not found, the correct error must be returned (e.g., FILE NOT FOUND vs. 
PATH NOT FOUND) 

• If OPEN_TARGETJDIRECTORY is specified, EFSD must request an open for the target directory from ECM, and 
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confirm its existence. The FileObjecfs name (from the IRP) must also be munged as specified above. 

• The Share Access value sent in must be checked against the existing one for the FCB if this is not the first stream 
associated with this FCB. 

• If any error is found, all data structures allocated must be released. 

• The IsFastloPossible member is set to FastloIsPossible. 

Cleanup 

There may be async posting of Cleanup requests. Some points: 

• The global resource and the FCB Resource must be acquired exclusively, posting if necessary 

• This represents the end of processing for this stream, but not a close of the file. 

• If caching is on, the cache must be flushed, and the pages purged. 

• The count of open handles in the FCB must be decremented 

• Any time stamps must be updated if accesses were done using fast I/O. 

• The F0J3LEANUP_C0MPLETE flag in the FileObject must be set 

• IoRemoveShareAccess() must be called 

Close 

There may be async posting of Close requests. 

• The global resource must be acquired exclusively 

• The CCB must be deallocated 

• The file's time stamps must be updated if the CCB bits indicate access/modification 

• If this is the last reference to the FCB for the file, the FCB is deallocated 

o any updated file attributes must be pushed back to the ECM in this case 

• If the file is marked to be deleted on close, the ECM must be asked to delete the file 

lead 

leads will definitely be open to async posting. Some points: 

• Non-buffered reads need to go straight to the ECM to request data; buffered reads must request data from the cache. 

• For now, we'll use the standard copy interface for cache reads 

o CcReadMdlQ for MDL reads 
o CcCopyReadO otherwise 

■ Note: the buffer to use might be for an allocated MDL, or it might be the UserBuffer! 

• If this is non-paging, non-buffered, and the file stream is also open for buffering, need to: 

o grab the PagingloResource exclusive 
o purge the cache for the range of the FCB 
o release the PagingloResource 

• Synchronization: grab the main Resource shared for non-paging reads; else the PagingloResource shared. 

• Zero length reads always return success 

• Reads starting beyond EOF return EOF 

• Requested lengths will be truncated to size if the request goes beyond EOF 

• The cache map must be initialized if this is the first buffered read for the FCB 

• For synchronous, non-paging reads, update the CurrentByteOffset in the CCB 

• For non-paging reads, a bit in the CCB must be set to indicate that the file was accessed 

^rite 
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• Non-buffered and buffered writes are as above for reads. 

• If this is a buffered write, need to call CcCanIWrite(); if false, call CcDeferWrite() and post for async processing. 

• A non-paging, non-buffered write for which the file stream is also open for buffering means: 

o grab the PagingloResource exclusive 

o purge the cache for the range of the FCB 

o flush the cache as well (Note: additional reqt!) 

o release the PagingloResource 

• Writes of length zero immediately succeed 

• A byte length of Low = FILE_WIUTE_TO_END_OFFILE, High = Oxffffffff signifies to start at EOF 

• For paging writes: 

o No write requests beyond EOF will be honored 

o If the starting offset is EOF or beyond, just return success 

o If ending offset is beyond EOF, truncate write length to EOF 

• For buffered writes: 

o If the write will extend the file size, inform the cache manager about the size change 

o If this is an MDL write, use CcPrepareMdlWriteO, else use CcCopyWrite(). 

o Again, as for reads, even for a non-MDL write, there may be an MDL allocated that we should use. 

• For synchronous, non-paging writes, update the CurrentByteOffset in the CCB 

• For non-paging writes, set a bit in the CCB specifying that the file has been modified. 

Fast I/O Read 

Initially at least, well just set the fast I/O read routine to FsRtlCopyReadO- 
Fast I/O Write 

Initially at least, we'll just set the fast I/O write routine to FsRtlCopyWrite(). 
Fast I/O Query Basic Info 

ITiis will just fill in the basic info buffer with the data in the FCB. 
Fast I/O Query Standard Info 

rhis will just fill in the standard info buffer with the data in the FCB. 
Fast I/O Query Open 

rhis will just fill in the network open info buffer with the data in the FCB, if the file exists. Some empirical observations 
Ve made using NTFS: 

• Regardless of whether the file exists or not, this will return TRUE (all fast I/O routines are boolean) 

• If the file does not exist, it will set the EOF size in the buffer to 0. The AllocationSize must be non-zero. All other 
fields seem to be don't cares. 

• If the file exists but is zero length, then both EOF and AllocationSize will be 0. 

• The IRP sent to this routine is for an IRP_MJ_CREATE; we can use more than just the name to identify the file, but 
also the security characteristics or whatever else is sent in the IRP. 

r ile Query Info 

Itandard queries will be supported; these however will not: 
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• Filelnternallnformation - no OPEN_BY_FILEJD 

• FileEalnformation - no EA data 

• FileCompressionlnformation — no on-disk compression 

• FileStreamlnformation — no multiple streams 

File Set Info 
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These actions will be supported: 

• EndOfFile size changes 

• Time stamp changes 

• File position changes 

• File disposition changes - delete pending 

• File rename requests 

In other words, all standard file set requests will be honored except AllocationSize changes. 
Directory Query 

This is an ugly NT interface, and unfortunately one that we need to expose across the ECM interface as well Some 
points: 

• These requests come in from the I/O Manager in a context-sensitive sequence. I.e., a request will come for the 
initial N directory entries; the next request will be for the next M entries; etc. Kind of like strtokO- 

• Thus, state must be maintained from request to request. This state will be kept in the CCB for a file stream, and 
consists of: 

o Pattern sent in on first request 

o Index of n'th entry to start retrieving with 

• My experience is that the INDEX_SPECIFIED flag is never set in a directory control query, even on queries 
subsequent to the initial one. 

• Here's the algorithm as best I can grok it from the fastfat sources: 

o If the CCB pattern field is empty, and the CCB flag specifying "match all" isn't set, this is the initial query 
o For the initial query, the main FCB Resource must be acquired exclusive, else we must acquire it shared. 

WHY? If we're only modifying the CCB, why lock the FCB? 
o The user has specified an index if the SL JNDEX_SPECIFIED IRP flag is set; we're ignoring all but the first 

match if SL_RETURN _SINGLE_ENTRY is set. 
o If this is the initial query, parse the pattern. If"*", set the "match all" flag in the CCB. In any case, save the 

pattern in the CCB. 

o Start with the current index in the CCB; for the initial query, this is 0; for subsequent queries, this is the index 
saved in the CCB. Both of these are overruled by the SL JNDEXSPECIFIED flag and value. 

o Both are also overruled if the SLJUSSTARTJSCAN flag is set; index goes back to 0. 

o After filling the values for an entry into the supplied buffer, update the CCB index field. 

o Fill the input buffer with as many entries as possible. If we run out of space, stop, and return 
STATUS_SUCCESS. 

o If there isn't space for the very first entry for this query (base length of record + filename) return 

STATUSJBUFFER_OVERFLOW. 
o Total number of bytes written to user buffer is returned in Status.Information. 
o If there are no entries in the initial query, return STATUS_NO_SUCH_FILE. 
o If there are no entries in subsequent queries, return STATUS JSTO_MORE_FILES. 
o The Filelndex field of the records returned in the buffer must be fixed up to be the byte offset from this 

record to the next record in the buffer. The Filelndex of the last record returned in the buffer should be 0; 

however, it appears from the fastfat sources that this isn't the case; they update the Filelndex value to the next 

entry offset, even if this can't fit in the buffer! This actually makes it a lot easier. 

• In order to satisfy this EFS interface, we need a relatively similar interface between the EFSD and the ECM. 
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o Ym proposing that only a single directory entry at a time be transferred between ECM and EFS. 

o We can further simplify things by essentially delivering only a FileBothDirectorylnformation buffer from 

ECM to EFS. This is a superset of anything the I/O Manager will request of EFS. 
o Someone has to do pattern matching: either EFS, ECM, or the eStream server. It shouldn't be the server. I'm 

proposing it be ECM. 

• Hence, the interface from EFS to ECM for directory queries could be: 

o Input: 

■ directory handle 

■ pattern 

■ starting index 

■ buffer 

■ buffer length 
o Output: 

■ (filled up buffer) 

■ number of bytes filled in to buffer 

■ if # bytes returned is 0, return code indicates either: 

■ no entries available 

■ an entry is available, but the buffer isn't large enough 

• This should evenly divide the work on the client machine between the EFS and the ECM, and allow the ECM code 
to be more portable than if all the complexity were pushed onto it. 

File System Query Info 

Empirically, I've noticed that the LanMan redirector returns failure for most of these requests. So, except for any user- 
iefined FSCTL requests we want to define, I'm going to fail all of these until it turns out we need to do otherwise. 

File System Set Info 
)itto for this IRP type too. 
Volume Query Info 

Ve at least need to minimally implement these requests: 

• FileFsAttributelnformation 

• FileFsVolumelnformation 

• FileFsDevicelnformation 

'ohime Set Info 

Ve will fail all requests of this type, 
lush Buffers 

i buffer flush request for a file stream will mean the following: 

• If the file stream isn't buffered, return immediately 

• The FCB main Resource is acquired exclusive 

• The Cache Manager is told to flush the buffer for the byte range of the file 

• The resource is released 
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A buffer flush for a directory is a successful NOP. 

System Shutdown 

IOCTLs 
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Functionality 

The Client Network Interface (CM) provides the interfaces for sending messages to serv- 
ers and provides threads for receiving responses and dispatching them appropriately. It 
uses the eStream Messaging Service (EMS) APIs to send and receive various messages to 
and from the application servers and SLiM servers. 

The number of threads in the CNI will depend on the functionality available from the 
EMS: In particular, more threads are necessary if the EMS provides asynchronous mes- 
saging capability (and the CNI uses this interface). The interfaces presented by the CNI 
are identical for both cases, but the internal organization of the component is not. 

The prefetcher will make calls to client networking interfaces (indirectly through ECM- 
ReservePage) to send requests for pages. Similarly, the LSM will make calls to acquire 
access tokens and subscription information. 

The networking component is responsible for examining the stream of requests to it and 
deciding when to coalesce multiple page requests into a single request to the server. 

The EMS does not provide reliability in the event of server failure. The CNI is responsi- 
ble for handling server failover and reissuing failed requests on different servers. The 
CNI abstracts the servers from other parts of the system. Clients of the CNI don't need to 
specify a particular server to make a request. 

Since the client networking component is where timeouts and retries occur, it is the com- . 
ponent that controls the policies for how long we wait for a connection to time out and 
how many times we retry a request before giving up. These parameters will be tunable. 
Any other parameters of the CNI that make sense to tune will be tunable. 

The CNI is also the component responsible for implementing the server selection policy. 

Data type definitions 

The CNI uses the request structure defined by the ECM. 

The CNI maintains an internal queue of messages that must be sent to servers. This 
queue is not exposed outside of the CNI. Like the ECM request queue, this queue will be 
maintained as a circular, doubly-linked list. 
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typedef struct NWRequest 

{ 

NWRequestType type; 

union {} parameters; /* params, depends on type */ 
struct _NWRequest *ne^t; .. . >r - 
struct _NWRequest *prev; 
} NWRequest; 

typedef enum 
{ 

CNI_PAGE_READ, 
CNI_ACQUIRE_ACCESS_TOKEN, 
CNI_GET_LATEST_APP_INFO, 
CNI_RENEW_ACCESS_TOKEN / 
CNI_RELEASE_ACCESS_TOKEN , 
CNI_REFRESH__APP_SERVERJ3ET , 
CNI_GET_SUBSCRIPTION_LIST 
} NWRequestType ; 

The CNI provides an enumeration of the parameters that can be tuned. This enumeration 
is expected to grow as the number of tunable parameters grows. 

typedef enum 

{ 

CNI_NUM_RETRIES , 
CNIJTIMEOUT, 
CNI_PROXY_ADDRESS , 
CN I_EFFECT I VEJBANDW I DTH 
} NWTunableParameter; 

r 

Related Components 

Theprefetcher and LSM call on the CNI to send requests to the app and SLiM servers. 
The CM makes calls to the ECM and LSM to inform them of responses that have come 
back from the server. The CNI will also make calls to EFSD interface functions when 
pages come back that satisfy EFSD requests. 

Interface definitions 

CNIGetPage 
eStreamStatus CNIGetPage( 

IN AppHcationID app, 

IN EStreamPageNumber page 

); 

CNIGetPage is the interface used by the ECM function ECMReservePage to request 
that a page be sent by the server. (ECMReservePage is called indirectly by the pre- 
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fetcher.) Note that no distinction is made between prefetches and demand fetches. To 
prevent race conditions or deadlock, the requested pages must already be marked as "in 
flight" in the index, and any requests for these pages from the EFSD must already be on 
the "in flight" queue before calling this interface. 

The CNI is responsible for selecting a server to direct this request to, and resending in the 
event of network or server failure. It will coalesce requests for multiple pages from the 
same application into a single request to the server. 

CNIGetSubscriptionList 
eStreamStatus CNIGetSubscriptionList( 

IN string Username, 

IN string Password 

); 

CNIGetSubscriptionList enqueues a request to acquire a subscription list from a SLiM 
server. When the subscription list is returned by the server, the client response thread 
will notify the LSM of the returned data via a callback defined in the LSM document. 

CNIGetLatestApplicationlnfo 
eStreamStatus CNIGetLatestApplicationInfo( 

IN uin tl 28 SubscriptionID 

); 

CNIGetLatestApplication enqueues a request to get the latest application information 
for a particular app. When the server returns the result, the CNI will notify the LSM of 
the returned data via a callback defined in the LSM document. 

CNIAcquireAccessToken 
eStreamStatus CNIAcquireAccessToken( 

IN uintl28 SubscriptionID, 

IN string Username, 

IN string Password 

); 

CNIAcquireAccessToken will cause the CNI to contact a SLiM server to retrieve an ac- 
cess token. The CNI is responsible for issuing retries if no response is received for a re- 
quest. The CNI will call the appropriate LSM callback function when the data come 
back. 

CNIRenewAccessToken 
eStreamStatus CNIRenewAccessToken( 

IN AccessToken Token, 

IN string Username, 

IN string Password 

); 

CNIRenewAccessToken will enqueue a request for access token renewal. When the re- 
sponse comes back, the CNI will dispatch the returned data to the appropriate LSM call- 
back routine. 
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CNIReleaseAccessToken 
eStreamStatus CNIReIeaseAccessToken( 

IN AccessToken Token, 

IN string Username, 

IN string Password 

); 

CNIReleaseAccessToken will enqueue a request for releasing an access token. When 
the response comes back, the CNI will dispatch the returned data to the appropriate LSM 
callback routine. 

CNIRefreshAppServerSet 
eStreamStatus CNIRefreshAppServer( 

IN AccessToken Token, 

IN uint32 BadQOS, 

IN uint32 NoService 

); 

CNIRefreshAppServerSet will enqueue a request for refreshing the app server set. 
When the response comes back, the CNI will dispatch the returned data to the appropriate 
LSM callback routine. 

The client networking component will also have routines for getting and setting tunable 
parameters. 

CNISetParameter 
eStreamStatus CNISetParameter( 

IN NWTunableParameter type, 

IN void *value 

); 

CNISetParameter sets a parameter. The actual type of value is determined by type, 

CNIGetParameter 
eStreamStatus CNIGetParameter( 

IN NWTunableParameter type 9 

OUT void *value 

); 

CNIGetParameter queries the current value of a parameter. The actual type of value is 
determined by type. 

Component design 

The internal organization of the client networking depends on the mechanisms available 
from EMS. Internally, the CNI interface functions put requests on a queue, and one or 
more threads services these requests by using the EMS to send messages to servers. 
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Synchronous Server Calls 



If EMS only provides a synchronous messaging service, a single thread will be used to 
perform all necessary actions. The CNI interfaces will put appropriate requests on the 
network request queue. They will also wake up the network communication thread, if 
necessary. 

The network communication thread's job is relatively simple. When it wakes up, it per- 
forms the following tasks: 

- choose a set of requests to be coalesced and remove these from the request queue 

- retrieve a server set via LSMGetAppServerSet or LSMGetSLiMServerSet, and choose a 
particular server for this request 

- make a synchronous EMS call to send the request 

- dispatch the response to the appropriate LSM or ECM callback 

If the synchronous messaging mechanism becomes a performance bottleneck, we can 
have multiple network communication threads to increase concurrency. 




(From prefetcher) 
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Asynchronous Server Calls 



The asynchronous case is a little bit more complex. Because of the proposed asynchro- 
nous call architecture, the client NW requires three threads. The CNI interfaces work just 
as they do in the synchronous case. They put requests on the network request queue, and 
wake up the network send thread. However, the actions performed by the CNI f s worker 
threads differ in the asynchronous model. 
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The network send thread is periodically awoken, and it coalesces requests off the NW 
request queue and sends them to the server. Unlike in the synchronous model, this thread 
does not synchronously wait for the request to come back from the server. Instead, it 
simply sends requests until the queue is empty, then goes back to sleep. 

The network receive thread waits for responses to come back from any server. Because 
of the EMS's asynchronous call implementation details, this thread posts returned data to 
a queue of responses to be handled by another thread. The network receive thread is also 
responsible for handling timeouts and reissuing those network requests on different serv- 
ers. 



Finally, the response dispatch thread pulls responses off the response queue, and handles 
the work of dispatching them appropriately. 
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Handling Network Failure 

When the client networking component is notified of a message failure by the EMS, the 
client worker thread will attempt to reissue the request on a different server. 

Coalescing Multiple Requests 

The CNI will coalesce multiple page requests that come from the LSM into a single re- 
quest to an application server. Multiple pages requests for the same application may be 
coalesced. No other types of requests may be coalesced, including page requests for dif- 
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ferent applcations. The CNI will not produce requests larger than the maximum allowed 
by the application server. 

Handling Persistent Failures 

There will be some persistent failures that will result in the network being unable to ful- 
fill page requests in a timely fashion. This may be due to network or server failure. 
(These may be indistinguishable from the CNI's point of view.) When the CNI has failed 
to satisfy a request for a certain amount of time, it will need to ask the user if he wants it 
to continue retrying, or if it should let the application terminate. It will do this via the 
CUIAskUserYesNo() interface. The client software control panel should include an op- 
tion to always wait until the server is available, and never ask the user if he wants the ap- 
plication terminated. 

Testing design 
Unit testing plans 

The testing harness for the networking component will be a set of dummy EMS drivers 
and a dummy NW client. The dummy EMS driver will be capable of performing a vari- 
ety of actions, including returning appropriate responses, returning inappropriate re- 
sponses, and timing out without any response. The dummy NW client will have knowl- 
edge about the expected EMS behavior, and will verify that the data it gets back from the 
network component are as expected. 

Stress testing plans 
Failure testing plans 

The client NW is the sole component responsible for implementing server failover. In 
order to test this code, it is necessary to implement a server with predefined bad behavior. 
The server failure modes that must be tested include 

- server that accepts a connection on a socket but doesn't respond to any requests 

- server that closes the socket before sending a response 

- server that closes the socket in the middle of a response 

- server that sends a partial response and then just stops 

- server that satisfies n requests then closes the socket or refuses to service more 

It is important that we cover scenarios that look like network failures and ones that look 
like server failures. (Are there other failure modes that are interesting?) 

Cross-component testing plans 

Cross-component testing of the client NW includes integration testing with the EMS, the 
LSM, and the prefetcher. Testing with the EMS can be performed in a manner similar to 
unit testing in conjunction with a specially written server. Testing with the LSM or pre- 
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fetcher can be performed in isolation by writing drivers for either the LSM or prefetcher, 
and using a dummy or real EMS. I'm not sure if this sort of testing is worth the effort to' 
write the appropriate harnesses. Verifying the output of such a combined system is cer- 
tainly trickier than testing any component in isolation. 

Open Issues 
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Functionality 

The eStream cache manager implements much of the client-side functionality for 
handling the eStream file system. The cache manager handles all file system requests 
made by the operating system by reading information from the cache or by passing the 
requests along to the profiling and prefetching component to fetch missing data from the 
network. 

The cache manager will initially be implemented in user space, but it may be useful to 
migrate it to the kernel for improved performance. In user space, it will be part of the 
eStream client process. In the kernel, it will probably be a device driver distinct from the 
eStream file system driver. 

The cache manager manages the on-disk cache of file system data, and the in-memory 
data structures for managing this cache. It does not manage prefetching of data from the 
server; that is the role of the eStream Profiling and Fetching (EPF) component. A 
separate networking component handles the network traffic. This component will also be 
described separately. 

Since there is no overall discussion of the client architecture at a more detailed level than 
the high level design, this document will cover that as well. 

Multiple cache page files will be supported. Each cache page file may be up to 2 GB in 
size. Different cache files may reside on different or the same logical disk (i.e. Windows 
drive letter.) 

Data type definitions 

An application ID uniquely identifies an eStream application. Just what constitutes "one" 
eStream application is not entirely defined, but different "builds" of the "same" app will 
be considered different eStream applications. For example, the Chinese-language version 
of Office is a different eStream application than the English-language version. 

typedef uintl28 ApplicationID; 

The eStream page number is the data type used to describe a page number within a 
particular file. Note that this is a page offset, not a byte offset. For eStream 1 .0, the 
cache manager will only support 2 GB cache files. 

typedef uint32 EStreamPageNumber ; 
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The fileld is used to uniquely identify a file within the universe of all eStream files across 
all eStream applications. 

>Pv A , ... „... * 

typedef struct { 

ApplicationID App, 

int32 File 
} fileld; 

The eStream page size is the fundamental size for eStream requests. This size is in bytes, 
^define ESTREAM_PAGE_S I ZE 4096 

The eStream file system uses the file time format of the Windows operating system. If 
the client runs on a system with a different native time format, the client software will be 
responsible for translating between the native format and the eStream format. The 
Windows data format is a 64-bit counter of the number of 100-nanosecond periods since 
January 1, 1601. 

EStream metadata is the file information supported by the eStream file system. This 
metadata is independent of the client or server operating system. 

typedef struct 

{ 

uint64 CreationTime; 
uint64 AccessTime ; 
uint32 FileSize; 
uint32 FileSystemAt tributes; 
uint32 EStreamAttributes ; 
} Metadata; 

The eStream inode contains the layout of a file in the cache. Each inode has the 
following structure: 

typedef struct 

{ 

Fileld Id; /* ID of this file; search parent for 
name*/ 

Metadata Metadata; 

FilelD Parent; /* parent directory 1 s file id */ 
uint32 NumPages; 
Pagelnfo *Pages; 
} EStreamlnode; 

The Pagelnfo array is variable sized. There is one entry in the pages array for each page 
in the file (not for each page cached, since we need to know whether the pages are 
present or not...) Note that the inode is only used in the "robust" implementation. 
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typedef struct 

{ 

EStreamPageNumber CachePageNumber ; 
PageStatus Status; 
unsigned char Priority; 
PageChecksum Checksum; 
} Pagelnfo; 

The page number doesn't require the 32 bits, since pages are 4096 bytes long. The extra 
bits will be used to encode which cache file this page resides in. The priority field is a 
number representing this page's priority for being kicked out of the cache. How exactly 
this field is used hasn't yet been determined. The checksum is a (fast) page checksum 
that can be used to validate the contents of this page. Note that it will be useful to have a 
slower, more effective checksum for development and a faster (but less thorough) 
checksum for deployment. 

The page status is an enumeration for the page's locking status (these are described in 
more detail later: 



typedef enum 

{ 

PS_INVALID, 
PS_CLEAN_UNLOCKED , 
PS_CLEAN_LOCKED , 
PS_DIRTY_UNLOCKED , 
PS_DI RTYJLOCKED , 
PS_IN_FLIGHT 
} PageStatus ; 



Note that this describes the layout of the tables in memory; how these data structures are 
represented on disk is described later. 

The EFSD file handle is a small integer passed between the EFSD and the ECM. This is 
used opaquely by the EFSD and is used as an index into an open file table by the ECM. 

typedef uint32 EFSDFileHandle; 

The ECM request type specifies the request type to the rest of the system. Note that som< 
"requests" are used to inform the prefetcher about the events handled solely by the ECM 
and do not actually request that any particular action be taken by the prefetcher. 

typedef enum 
{ 

ERT_READ, 
ERT_WRITE, 
ERT_READ_H I T , 
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ERT_WR I TE_H I T 
} ECMRequestType; 

The ECM request is a request descriptor that is used in various lists within the cache 
manager. These lists are doubly-linked, circular lists. 

typedef struct _ECMRequest 

{ 

uint32 RequestID; /* same as EFSD request id */ 
ECMRequestType RequestType; 

union {} Parameters; /* union of all parameters*/ 
struct _ECMRequest *next; 
struct _ECMRequest *prev; 
} ECMRequest; 

The cache manager must maintain an array of files that have currently been opened by 
the EFSD. This array will be statically allocated. This will put a limit on the number of 
files that may be opened concurrently on the eStream file system. The elements of the 
array are the following: 

typedef struct 

{ 

uint32 Valid; 
fileld File; 

HANDLE OpenFile; /* for simple implementation */ 
eStreamlnode *Inode; /* for robust implementation */ 
} OpenFilelnf o; 

The cache manager maintains a hash table containing information about each application 
that currently has open files. The hash table is indexed by app ID, and contains the 
following active app information records: 

typedef struct 

{ 

AppID App; /* identity of this app */ 
uint32 OpenFiles; /* # of open files */ 
uint32 HaveAccessToken; /* boolean */ 
} ActiveAppInf o; 

The ECM will use this table to quickly determine whether it should continue processing a 
request it gets from the EFSD, or if the request should be passed to the LSM to ensure 
that an access token is available. See the section below on ECM-LSM interaction for 
more details. 

The LSM uses the access token state to specify a state for an access token. Right now, 
we only plan to support valid and invalid, but it may be interesting in the future to allow 
already opened files to be read, but no new files to be opened. 
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typedef enum 
{ 

ATS_ I NVAL I D , 
ATS__VALID, 
ATS_VALID_NO_OPEN 
} AppTokenS t a t e ; 



Interface definitions 

The ECM exports the following interfaces for operating on the cache. They may be 
called by the cache manager, prefetches or networking component. (Not all components 
are expected to call all interfaces; see each interface description for more details.) 

Note that the cache interfaces are defined at a very high level as the actions that may be 
performed on the cache by the components, such as enqueuing a new request. They have 
been defined this way so that these intrinsic operations can be implemented correctly 
once and limit the possibility that an individual component will not perform proper 
actions. 

ECMReservePage 
eStreamStatus ECMReservePage( 
INfileldF/fe, 

IN EStreamPageNumber Page, 
IN ECMRequest ^Request 

); 

ECMReservePage reserves a page in the cache for a request. This interface is called by 
the prefetching component, and will send a request to the network component. Logically, 
this interface reserves an empty cache page for this request (if one is available), puts this 
request on the "in flight" queue, and calls on the network to request the page (unless it is 
already in flight.) 

ECMIsPagelnCache 

eStreamStatus ECMIsPageInCache( 
IN fileld File, 

IN EStreamPageNumber Page 

); 

ECMIsPagelnCache returns TRUE if the specified block is in the cache, and FALSE 
otherwise. It is used by the EPF to determine if it should prefetch a block; normally, the 
EPF would choose not to prefetch something that is already in the cache. Note that it 
would be a good idea for the prefetcher to adjust the priority of a page that it thinks it 
wants to prefetch, so thai they are less likely to be evicted from the cache before they are 
needed. 

ECMDepIanePage 
eStreamStatus ECMDeplanePage( 
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IN fileld File, 

IN EStreamPageNumber Page, 

IN char Buffer [ESTREAM_P AGE _S1ZE] 

)>• 

ECMDeplanePage performs all the necessary actions for writing a page coming off the 
network into the cache and back to the EFSD. This consists of copying the page into the 
cache, remove all pending requests for this page from the in flight list, marking the page 
as clean/unlocked, and returning the page to the EFSD for each in flight request. 

ECMReadPage 
eStreamStatus ECMReadPage( 
IN fileld F//e, 

IN EStreamPageNumber Page, 
IN ECMRequest ^Request 

); 

ECMReadPage performs all the necessary actions for attempting a page read from the 
cache. The cache is checked to see if it contains the page; if so, the page is copied to the 
buffer, the EPF is notified of the hit, and appropriate status is returned. Otherwise, this 
page is put on the queue for requests pending to the prefetching component, and ' 
appropriate status is returned. 

ECMWritePage 
eStreamStatus ECMWritePage( 
IN fileld File, 

IN EStreamPageNumber Page, 
IN ECMRequest ^Request 

); 

ECMWritePage performs all the necessary actions for attempting to write a page in the 
cache. Note that this could be somewhat more complex than a read, because a partial 
write to a page might necessitate reading the page from the server before writing the 
partial page to the cache. 

The following interfaces are the abstract interfaces that the ECM will use to communicate 
with the EFSD. Hiding the EFSD's raw DeviceloControls behind these interfaces will 
help make porting the ECM into the kernel easier, should we decide to do that. 

ECMSetTokenState 
eStreamStatus ECMSetTokenState( 

IN Appld App, 

IN AppTokenState State 

); 

ECMSetTokenState is called by the LSM to indicate to the ECM that a token has 
become available or has expired. The main effect of this interface is to update the state of 
the speofied application in the active app table. See the ECM-LSM interaction below for 
more details. 
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ECMGetCachelnfo 
cStreamStatus ECMGetCacheInfo( 

OUT UNICODE_STRlNG Location, 

OUT uint32 * CurrentSize, 

OUT uint32 * MaximumSize 

); 

ECMGetCachelnfo is called by the client user interface to find out where the ECM 
cache is located and its current and maximum size. Location is an absolute path name of 
the cache file. 

ECMSetCachelnfo 
eStreamStatus ECMSetCacheInfo( 

IN UNICODE_STRINGIoca//o«, 

IN uint32 MaximumSize 

); 

ECMSetCachelnfo is called by the user interface when a new cache location or size has 
been requested. Note that the cache manager may only begin using the new cache 
information after a restart of the client software (which may only occur on client machine 
reboot.) The client UI will call this interface when it wants to make a change; the ECM 
is responsible for actually resizing the cache and making any changes necessary to 
persistent storage (i.e. the registry). 

EFSDGetRequest 
eStreamStatus EFSDGetRequest( 

OUT EStreamRequest ^Request 

); 

EFSDGetRequest reads the next request from the EFSD, including any parameters that 
need to be passed. This may involve one or more DeviceloControl calls to the EFSD. 
EFSDGetNextRequest is responsible for allocating memory for this request, and an 
EFSDCompleteRequest call will be repsonsible for deallocating the memory. 

EFSDCompIeteRequest 
eStreamStatus EFSDCompleteRequest( 

IN EStreamRequest *Request, 

IN ECMErrorCode Status 

); 

EFSDCompleteRequest will be called for each request that is received by the ECM via 
EFSDGetRequest. status indicates the completion status for this request, and may 
indicate success, a retry, or a particular failure condition. Non-persistent errors will be 
handled by the ECM internally or by requesting a retry of a particular request. Errors 
reported to the EFSD will be propagated up the file system stack. 



Overall Client Architecture 
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The eStream client will have various types of threads in order to perform its work. The 
basic architecture is illustrated by the following diagram. 



Get access token, release access token 



Token acquisition notification, token expiration notification 
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The FSD worker thread will pull requests from the FSD. It will return data for requests 
that can be satisfied immediately. Any request that requires information that is not 
currently in the cache will be put on a queue for the prefetching thread to handle. 

The profiler will receive all cache misses from the FSD worker thread. Using its own 
data structures (which may include information about recent cache misses in addition (o 
information about general prefetch patterns), it will decide which blocks it should 
prefetch. Demand fetch and prefetch requests are sent to the network component. The 
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only way demand fetches and prefetches are treated differently by the network 
component is that demand fetches are sent to the EFSD while prefetches are not. 

The network thread will manage open connections to app servers and retry requests that 
time out. When data comes back from the network, the network thread will copy the 
returned buffer into the cache and to the FSD, if the request was a demand miss. 

The cache manager consists of the EFSD worker thread and the APIs to access the cache 
index, the data blocks, and various queues used by threads in the client. 

Not shown on the diagram is an error thread. This thread is responsible for calling the 
client UI module indicating appropriate error messages and waiting for the user's input. 
When any component decides that it has an error condition that requires user input, it 
calls ECMReportError with the request and an appropriate error condition, which will 
be enqueued for the error thread to handle. For example, when the network interface 
times out reading a page from an application server enough times, it will call 
ECMReportError. When the error thread gets to this request in the queue, it will ask 
the user if he wants to wait until the app server is available or allow the application to 
terminate. 

ECM-LSM Interaction 

The ECM-LSM interaction is a relatively simple one. The LSM notifies the ECM when 
it first receives an access token and when its access token expires. It does this via the 
ECMSetTokenState interface. The ECM keeps track of each application that has had 
files open, and whether or not we have an access token for each of these apps. 



App ID 


# of open files 


Have access token? 















Note that the LSM need not notify the ECM of mundane events like renewals as long as 
some token is valid. Also, the ECM does not keep track of the token itself, just whether 
or not we have a valid one. An additional nicety of this approach is that we could allow 
the ECM to satisfy requests out of the cache as if we have an access token, without 
actually having one. 

When it receives a request, the ECM checks its table to determine if an access token is 
available. If it is, it handles the request as normal. If not, it asks the LSM to acquire an 
access token via LSMGetAccessToken. The LSM may return that it has a token, in 
which case the ECM will continue to process the request, or the LSM may say it doesn't 
have a token, in which case the LSM takes ownership of the request and will reissue the 
request when the access token is available. 
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When the number of open files drops from 1 to 0, the ECM will mark the token as invalid 
in its table and call LSMReleaseToken. The LSM may choose not to renew access 
tokens that have been released. 

Component design 

Two cache organizations will be presented. One is suitable for a quick implementation 
but doesn't lend itself particularly well to high performance or easy manageability; the 
other will be more difficult to implement but should provide better performance. I will 
first describe some data structures that are shared by both designs, then go into the 
specifics of each design. 

Common Data Structures and Algorithms 

Certain request lists are common to both cache organizations. One is a queue between 
the FSD worker thread and the prefetching thread for demand fetches that have not yet 
been seen by the prefetcher. The other is a list of all requests for pages that are "in 
flight." Requests from the in flight list are removed when they have been satisfied. The 
in flight list is unsorted and searched whenever a request comes back for requests that 
match the returned page. If the performance of this data structure becomes an issue, we 
will change its organization for faster lookup. 

Both request lists use the request data structure described above. 

The ECM will maintain an array of files currently opened by the EFSD. On file opens, 
an empty location in this table will be allocated for the newly opened file, and the index 
to that entry returned as the file handle. (Note that the way the interface between the 
ECM and the EFSD is defined, it is an error to open an already opened file. The cache 
manager will have to detect such cases and report an error, but it will not keep a reference 
count of the number of opens on each file.) This mechanism will allow the ECM to keep 
track of the volumes that currently have opened files as well as abstracting the 
client/server file ids away from the kernel driver. (This might allow us to update the 
client/server protocol without rewriting the EFSD.) 

Easier Implementation 

The cache will be implemented as a directory tree on the user's hard drive that parallels 
the eStream file system. Each file will contain a header and an array of status bytes in 
addition to the data blocks that the file contains. The array of status bytes has one byte 
for each page in the file. Each byte indicates the current status of that page in the file. 
(Pages have several different states, so a simple bit per page is not sufficient.) Each file 
will thus look like 



Header 



Page Status Bytes 



File contents page 0 
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File contents page 1 



The header is defined as: 
typedef struct 

{ 

uint32 magicCookie; 

uint32 headerLength; /* Length of this header, in bytes */ 

fileld fileid; ■/* for sanity checking */ 

uint32 length; /* Length of the file, in bytes */ 

uint32 firstPage; /* Offset to the first page in the file 

*/ 

Metadata metadata; 

} BCMCacheFileHeader; 

The page status bytes begin immediately following the header, and this area is padded 
with zeros to a page boundary. The first page of the file's contents (and thus each 
following page of file contents) will therefore begin on a page boundary. 

Note that one issue with this design is that files that approach the file size limit of the 
underlying file system cannot be represented, due to the overhead with the header and 
bitmap. If this design is used solely for early engineering efforts, then this limitation is 
acceptable. If we have to work around this limitation, one way to do it is to make the 
headers and page status bytes reside in a separate file or files. 

Directory contents would reside in server format in a file named "Directory" inside of the 
directory whose contents they represent (with the addition of the header and status bytes 
as described above for ordinary files). For example, z:\Program Files\Microsoft Office 
would reside in c:\Cache\Program FilesVMicrosoft Office\Directory. This has the 
drawback of creating special file names that can't be used by files in the eStream volume, 
but again, for an early engineering implementation, this is an acceptable limitation. 

Another issue with deploying this implementation is that it is trivial to reverse-engineer 
this file format and copy files directly from the cache. 

Robust Implementation 

The cache will be organized into an index file and one or more cache data files. Multiple 
data files may be necessary as we may wish to allow the cache to grow larger than the 2 
GB file size limit (for some native file systems) or to span multiple drive letters on the 
client. The data files will only contain pages of file content. These pages will be aligned 
on page boundaries. The index file contains all the information needed to locate file 
pages, and is contained in a separate file for simplicity. 

Page and index files must reside on a local disk (rather than a network disk) and cannot 
be shared by multiple clients. 
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Each file with any pages currently resident in cache will have a data structure containing 
information about that file, including its file id, the file id of the directory containing it, 
the file's metadata, and the map for finding the file's data blocks. This data structure is 
very similar to the inode of a traditional file system, and will be referred to as the eStream 
inode. A naive implementation of the inode is described above; no doubt, we will want 
to reorganize this data structure for more compact representation and better performance. 
Note that one requirement of the inode is that it contain a status field for each page in the 
file. One character is sufficient for this status; whether or not we can make do with 
fewer than 8 bits is an open question. 

A hash table will be used to map file IDs to file inodes. 



Index hash table 



Collision chains of file inodes 



"too" inode 



"bar" inode 




"baz* inode 




The inode contains pointers to each block's location in one or more cache page files: 
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Foo's Inode 



Cache page file 1 Cache page file 2 
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To prevent race conditions, a single lock controls access to both the hash index and the 
linked list of requests that are pending network access. Individual pages in the cache may 
be locked for read or write access. Since each page's status is in the index, the index 
must be locked order to lock a page for reading or writing. The page states are controlled 
by the following state machine: 



Omnishift Confidential 



Page 13 




The dirty/clean distinction is between those pages that we have written locally (and thus 
cannot evict from the cache) and those pages that we haven't written (and thus can be 
refetched from the server). 

A page would be locked while it was being read or written for copying to the file system 
driver. The operation may thus proceed with the index unlocked, without the possibility 
of page eviction while a copy is still in progress. The FSD worker thread is the only 
thread that reads or writes pages from the cache, so it's the only thread that can lock or 
unlock these pages. The in flight state is only for pages that are currently being fetched, 
either as a demand fetch or as a prefetch. The prefetching thread is the only thread that 
will put pages into this state, and only the networking thread will move pages from in 
flight to unlocked. 

A list will be maintained of all "in-flight" requests. A single lock will control access to 
both this list and the cache index, so there are no race conditions between items being put 
on this list and data coming off the network. When the FSD worker thread gets a request, 
it acquires the index lock and looks at the status of the page. If the page is clean or dirty 
but unlocked, it will lock the page and copy it to the FSD. If the page is invalid, then this 
is a demand fetch, and the request is forwarded to the prefetches If the page is marked in 
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flight, then this is either a second request for an outstanding demand fetch, or it is a 
request for an in flight page. Either way, while this thread still holds the index lock, this 
request will be inserted into the list of in-flight requests. Race conditions might occur 
because the FSD might make multiple demand reads of the same page, or it may make a 
demand read to a page that is already in flight due to a prefetch. 

Reading requested pages off the network and writing them to the cache (and to the file 
system driver, if necessary), are where this race condition comes up. We need to ensure 
that a request for a page that has arrived does not end up in the list of "in flight" requests. 
The solution is the following: When a data page comes back from the server, the 
networking component acquires the index lock to find the cache location of this incoming 
page. If the page is not marked in flight in the cache, this is a bug. (Of course, this is a 
relatively benign bug, and the NW component could just ignore the page.) The 
networking thread leaves the page as marked in flight, however, and unlocks the index. It 
writes the incoming page into the proper location, but it saves the in-memory copy of the 
page. It then reacquires the index lock, marks the page as clean/unlocked (since it's now 
in its final location in the cache), removes each request in the in-flight list for this page, 
then releases the lock. (Any further requests for the same page will find the page 
clean/unlocked, so the FSD worker thread will be able to satisfy these requests directly.) 
The networking component then proceeds to satisfy all of the requests it pulled off the in- 
flight list by using the copy of the page that it saved in memory. This way, it doesn't 
have to lock the index the entire time it is sending completed requests to the FSD. 

Each of these complex scenarios is captured in the cache file's API's. As long as these 
are implemented correctly, other components don't need to worry about the exact 
sequence of operations that needs to occur. 

Free Space Management 

Free pages will be maintained as a free list in memory and as a bitmap on disk. The free 
list will be built from the bitmap on eStream client software startup. Access to the free 
list will be controlled by the same lock controlling access to the index. 

Evicting Cache Pages 

Individual cache pages may be evicted. There is an 8-bit field in the index for each 
page's importance. Initially, we will implement a random page replacement policy. 
Later, we will use this page importance field in an unspecified way to replace pages in 
such a way as to maximize interactive user performance and minimize application server 
load. Only clean/unlocked pages may be evicted. Pages that are evicted will eventually 
be put on the free list. Page eviction will only happen at "garbage collection" time. See 
"crash resilience and garbage collection," below. 

Handling Cache Size 

Growing the cache should not be an issue. The cache manipulation routines must know 
the overall size of the cache, in pages. Increasing the size of the cache on the fly should 
be a relatively straightforward process, as we merely need to lengthen the cache file(s) 
and add the new pages to the free page list. 



Omnishift Confidential 



Paee 15 



Unfortunately, shrinking the cache is a much more difficult operation, since it potentially 
involves moving around pages that might currently be in use for paging operations or be 
in flight from the network. Changing the cache around at runtime is both difficult to 
implement correctly and a performance problem. The current plan is to support shrinking 
the cache only at eStream client software startup. The maximum allowed size of the 
cache will be stored in the Registry. On eStream client software startup, the current size 
of the cache will be compared against the allowed size specified in the registry; if it is 
larger than the maximum size specified in the registry, then the size of the cache will be 
reduced by evicting files and compacting the freed space. A request by the user to reduce 
the size of the cache will take effect the next time the client software starts. 

Note that files that the user writes to the z: drive are not considered candidates for 
eviction (unless the file is explicitly deleted.) This means that the user's on-disk cache 
may in fact grow to be larger than the limit they specify. 

Also note that at least one free page (not used by user-written files) is required for the file 
system to make forward progress. We also may want to require some minimal amount of 
cache before eStream will even run. Thus the maximum cache size specified by the user 
should be considered a "soft limit." There would be a "hard" minimum amount of space 
equal to the number of pages required to store the files written by the user on the z: drive 
plus a small amount of cache we designate just for running eStream. If this hard 
minimum is greater than the soft maximum specified by the user, the hard minimum 
would win. I would recommend preallocating and non-zero filling the file on disk so that 
we know that the space is available. 

Crash Resilience and Garbage Collection 

In order to provide crash resilience, the index will be periodically checkpointed to disk. 
Note that allocating blocks does not cause problems if the index is not updated. 
However, we cannot reuse a page's storage until that page has been marked free on disk. 

The solution to this problem is to periodically garbage-collect the cache (if it is nearly 
full), and writing the index to disk. The cache manager will alternate between writing 
two cache index files. The index file will have a marker at the end that indicates that it 
has been successfully written and a time stamp, and on startup the ECM will use the 
latest, fully written index. 

Data blocks will always be written directly to the cache page files. These files must be 
flushed before writing the index. 

Garbage collection involves the following steps: 

- lock the index 

- copy the free list 

- choose blocks in the cache to free, and make a list containing just the newly freed 
blocks. Mark these blocks as invalid in the file's inodes, but don't put them on the free 
list (yet) 
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- make a copy of the index 

- unlock the index 

- merge the list of newly freed blocks with the copy of the free list 

- flush all cache page files 

- write the new, merged free list (as a bitmap) and index to disk 

- lock the index 

- add the newly freed blocks to the free list 

- unlock the index 

- free any allocated data structures 

Index File Contents 

The index file contains the following items: 

- List of cache block files, with their sizes 

- Free block bitmap, per cache block file 

- Inodes for all files; may be stored hashed or may be rehashed on startup. 

Testing design 
Unit testing plans 

Cache file manipulation routines can be tested in isolation. We will write a standalone 
harness that exerc.ses the functionality of the cache file manipulation routines by 
performing cache level operations directly. A multithreaded unit test for the cache 
manipulation routines would be ideal, so we can test the correctness and performance of 
our locking strategy without the need to build the entire cache manager. 

Each "thread" of execution described by this document can be separately tested by 
creating a testing harness providing that thread inputs and monitoring its outputs 
Replacements for the EFSD interfaces can be very effective here. 

Stress testing plans 

An interesting stress test for the cache manager is if it can work correctly with very small 
caches even all the way down to 1 page. (Or at least, a cache with all pages but one 
marked as dirty.) v 6 

The cache manager will be able to operate in "verify mode," where requests that hit in the 
cache will still be sent to the server, and the pages returned by the server will be 
compared with the cached page's contents. 

The cache manager will support multiple different page checksum algorithms We can 
use a fast algorithm for deployment while using a more rigorous one in development 
This also has the benefit of allowing us to test the performance impact of various 
checksum algorithms. 
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The cache manager will have the ability to verify the integrity of the cache index and free 
page bitmap. In particular, it will have the ability to determine taht no pages are allocated 
to more than one file in the file system, and that each page belongs to a file or is on the 
free list. 

Stress testing for the ECM will include crash testing. 
Cache manager testing will include resizing the cache. 

Coverage testing plans 
Cross-component testing plans 

We can build a "cache only" file system by not using the prefetching and network 
components. This allows us to test the EFSD in conjunction with the cache manager 
without involving the prefetcher or the network component. 

Early implementation of the client will likely involve a null prefetcher that does no 
prefetching. 

We can use the testing harness for the cache manager that doesn't use the EFSD to drive 
the cache manager in conjunction with the prefetcher and network component. This 
allows us to test the combination of these components without driving it with the live file 
system driver. 



Upgrading/Supportability/Deployment design 

The client user-mode software and device drivers are packaged separately. (I.e. the client 
executable and the drivers are separate files on the disk.) This leads to the possibility of a 
"partial" upgrade that results in inconsistent versions of the drivers and client user-mode 
software. The drivers should support an interface that returns the version number of the 
driver, or of the interfaces provided by the driver. This will help the client software to 
recognize situations where it should tell the user to reinstall the client software and not 
result in bad system behavior. 

Most (all?) on-disk data files should have file headers containing at a minimum a magic 
cookie and the file format version number. This will help us with upgrades in the future. 



Open Issues 

We need to address what happens when a fetch is requested and no empty space can be 
found m the cache. The prefetcher should probably block until such time as space is 
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made available for this request. While operating with very small amounts of cache will 
obviously cause bad performance, it should not result in a deadlock. 
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eStream Cache Manager Straw Man Proposal 

Version 0.2 



Purpose 

The purpose of this document is to serve as the basis for the design of the eStream Cache 
Manager. As a straw man, this document is meant to serve as the basis for discussion, 
and anything here is subject to change. Assuming there are no major concerns with this 
document, I will proceed with producing a low level design for the cache manager. 

Requirements in Brief 

Support > 2GB client cache, possibly across multiple drives 

Provide some level of protection against piracy, via both the file system and the cache 
Fast lookup for what is in the cache and where to find it 
Support automatic and user- specified cache size policies 

As far as cache size goes, I think that it is reasonable for eStream 1 .0 for the cache to be 
limited to one disk partition and 2GB of space, but the design should allow for very large 
caches (spanning more than one file and possibly more than one drive letter.) Note that if 
the cache is greater than 2GB in size, it cannot be mapped into the address space of a 
single process under NT/2000 on x86. 

Cache Organization 

The cache will be contained in 2 or more files. One file will contain the cache indices, 
and one or more files will contain the data blocks for cached files. (More than one cache 
data file may be required if the cache is larger than the largest file allowed on the native 
file system.) This allows us to keep the cache index file memory mapped and only map 
the data file(s) if there is enough memory space to do so. 

Data Blocks 

The cache data file will contain data pages frome the file system 4k in size. 
Data will be stored in the cache uncompressed to allow easy page retrieval. 
Cache Index 

The cache index will be a b-tree. The key for the lookup will be the file id and page 
number requested. Keys in the b-tree are the set { volume #, file #, starting page, # of 
pages }. A lookup will succeed when the volume number and file number match, and the 
requested page is in the range from starting page to starting page + # of pages. The data 
stored for that key will be the offset into the cache for the beginning of the run. As is 
described in the file system proposal, the file number and starting pages are each 32 bits 
long. ] propose making the starting page a 48 bit number and the number of pages a 16 
bit number. This allows us to have a very large total cache and reasonable sized runs of 
contiguous pages in the cache. 



Free space in the cache will have to be managed. Free blocks can be placed into a 
specially identified "free space file" in the index. Some auxiliary data structures may be 
convenient to make searching for a regiori~of free* space of a particular size. 

Metadata for a file would be stored in the cache. It would be indexed by page number -1 
in the index. 

Cache Replacement Policy 

For simplicity, I propose that the cache manager evict entire files from the cache when it 
decides that it needs to clear room in the cache. (Of course, any fragmentary file that is 
in the cache can be evicted.) We should implement LRU for cache replacement, so we 
will evict files for apps that have not been run recently. 

One Cache Per System 

Administrator priviledges are required to install eStream. While various users on a 
system might have conflicting desires about eStream configuration, such as the size of the 
cache, I think that it is reasonable to have a policy where the adminstrator controls the 
setup of the eStream client. By limiting the cache to one per system, we eliminate any 
ambiguity about cache use in a multiuser environment. 

Profiling and Prefetching 

Profiling and prefetching have been broken out as a separate component in the client. It 
will be described elsewhere. It is expected that while the profiler/prefetcher will want 
access to the cache data structures (i.e. it wants to know what's already in the cache), the 
logic associated with prefetching is not logically tied to the cache manager, and should 
thus be separated. 

Future Directions 

Compression of the cache could potentially be a big win. We could provide cache 
compression similar to the way that NTFS provides file compression - we compress some 
number of blocks at a time (e.g. 16) and only store the compressed data when it saves at 
least one block of storage. Caching of data on disk can sometimes be a performance win, 
since decompressing the data can be faster than trasferring it on disk if the disk is slow 
enough. 
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Functionality 

The eStream Windows NT/2000 File System Driver (EFSD) is a kernel-mode file system 
driver to which file requests will be forwarded by the NT I/O Manager. It is the point of 
contact for users to access files on an eStream server. It works with the NT File Cache 
Manager to insure that kernel file caching is available for eStreamed files. 

The Windows 98 EFSD is almost certainly to be very different from the driver for WNT 
and Win2K, and will not be described here. 

In this document, I'm assuming that the EFSD communicates closely with the eStream 
cache manager (ECM) to perform the various file system requests. There may in fact be 
several components— if for example the ECM is broken into sub-components. Also, this 
document assumes that the ECM is in user mode; if this ends up in kernel mode, we will 
need significant changes to the interfaces to it. 

Data type definitions 

File handle 

A file handle passed between the EFSD and the ECM is defined by the ECM: 
typedef uint32 EFSDFileHandle; 

Names 

All file and directory names will be passed as counted Unicode strings, basically as de- 
fined by the NT header files. Note, however, that in NT the Buffer field is a pointer; for 
our purposes in communicating with the ECM, it's a NULL-terminated variable length 
array: 

typedef struct _UNICODE_STRING { 
USHORT Length; 
USHORT MaximumLength; 

USHORT Buffer[l] ; // NULL- terminated, 2-byte 

// characters 

} UNICODE_STRING; 
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Time stamps 

The NT standard time format is a signed 8-byte integer representing the number of 100- 
nanosecond intervals since January 1 ? 1601 . These time stamps will be tracked for files 
and directories: 

□ Creation time 

□ Modification time 

File attributes 

File attributes are contained in an unsigned 4-byte integer. This subset of attributes from 
Windows NT will be supported: 

FI LE_ATTRI BUTE_READONLY 
F I LE_ATTR I BUTE_DI RECTORY 
F I LE_ATTR I BUTE__ARCH I VE 
FILE_ATTRIBUTEJSJORMAL 
. F I LE__ATTR I BUTE_TEM PORAR Y 

These attributes are not supported: 

F I LE_ATTR I BUTE_H I DDEN 

F I LE_ATTR I BUTE_S Y STEM 

F I LE_ATTR I BUTE_DE VI CE 

F I LE_ATTR I BUTE_S PARS E_F I LE 

FILE__ATTRI BUTE_REPARSE_POI NT 

F I LE_ATTR I BUTE__COM PR ESS ED 

FILE_ATTRIBUTE_OFFLINE 

FI LE_ATTRI BUTE_NOT_CONTENT_I NDEXED 

F I LE_ATTR I BUTE_ENCR YPTED 

File size 

File size will be represented as a 4-byte unsigned integer. Since sparse files are not sup- 
ported, there will only be one file size passed between the ECM and the EFSD. 

Metadata 

This structure is defined to pass file and directory metadata between the EFSD and the 
ECM: 

typedef struct { // 24 bytes, 4-byte aligned 
int64 CreateTime; 
int64 Modif yTime; 
uint32 FileSize; 
uint32 Attributes; 
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} Metadata; 

Interface definitions 

The EFSD is called by several different components, including 

□ the NT Executive (I/O Manager, Virtual Memory Manager), for standard file sys- 
tem requests 

□ the ECM, for these same file system requests, and to invalidate cached pages for 
coherency 

□ the client start software, to start and stop the EFSD 

The EFSD supports standard FSD interfaces to the NT Executive modules; not all possi- 
ble interfaces are supported, because the eStream file system is relatively low- 
functionality (compared to NTFS, for example). 

The following file system requests will be supported; the interfaces for them will not be 
shown here, as they can be found in the DDK documentation. 

□ Create IRPs, for both new and existing files 

□ Cleanup and Close IRPs 

□ Read and Write IRPs: 

o synchronous and asynchronous 
o cached and non-cached 
o paging and non-paging 

□ Fast I/O reads and writes (with buffers or MDLs) 

□ File information (get and set) IRPs 

□ Directory query IRPs 

□ Volume information (get and set) IRPs 

□ File system information (get and set) IRPs 

□ Flush buffer IRPs 

□ System shutdown IRPs 

□ Various Fast I/O queries 

The EFSD will not handle Directory Notification IRPs, nor will it support hard links 
(which are supported natively on NTFS on W2000 only): neither of these requests are 
required, and no expected user functionality will be lost without them. We are presently 
not supporting byte-locks; this may need to be revisited if the need arises. 

In addition to the interfaces to the NT Executive, the EFSD will support various inter- 
faces from other client components; all these will be sent via IOCTL calls. The first ones 
listed are simple support interfaces; the interfaces between the ECM and the EFSD fol- 
low these. 

An IOCTL coming in to the kernel— via a DeviceIoControl() call— has the following pa- 
rameters: 
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□ IOCTL control code 

□ input buffer pointer 

□ input buffer size 

□ output buffer pointer 

□ output buffer size 

□ pointer to a 4-byte variable to receive the number of bytes written to the output 
buffer 

Q pointer to an OVERLAPPED structure for asynchronous operation (always 
should be NULL for EFSD) 



All of the following interfaces are described in terms of the IOCTL buffers sent and re- 
ceived for each control code. 



The following interfaces are called from the controlling client component (StartClient). 

Starting and stopping the file system 

The eStream FSD will be loaded into the kernel when a system is rebooted; i.e., it is al- 
ways resident. If applications access files on this FSD via a drive letter, then the file sys-- 
tem is implicitly turned off while a symbolic link for that drive letter is not present. Even 
when a drive letter symlink exists, the EFSD will not accept requests until the START 
IOCTL is sent. 



These IOCTL control codes will be defined for starting and stopping the eStream FSD: 

IOCTL_EFS_START_FS 
IOCTL_EFS_STOP_FS 

Starting the FSD 

The input buffer for the START IOCTL should have the following: 

□ version id: 4-byte identifier for the client component 

□ debug flags: 4-byte value indicating the debug level to use 

The output buffer for this IOCTL will be filled with the following: 

□ version id: 4-byte identifier for the EFSD version present 

□ status: 4-byte value, with one of the following: 



EFS_STATUS__SUCCESS 
EFS_STATUS_BAD_VERSION 
EFS_STATUS_BUFFER_TOO_SMALL 
EFS_STATUSJDUPLICATE_REQUEST 
EFS STATUS ABNORMAL TERMINATION 
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The status return value from this IOCTL will be one of the following: 

□ STATUS_SUCCESS 

□ STATUS_INVALID_DEVICE_REQUEST 

A DUPL1CATEREQUEST error is returned if the FSD is already started. 
Stopping the FSD 

The input buffer for the STOP IOCTL should have the following: 

□ force: 4-byte value 

o 0: shutdown only if no outstanding files are open 
o 1 : shutdown regardless of state of open files 

The output buffer for this IOCTL will be filled with the following: 

□ status: 4-byte value, with one of the following: 

EFS_STATUS_SUCCESS 
EFS_STATUS_BUFFER_TOO_SMALL 
EFS_STATUS_DUPLI CATE_REQUEST 
EFS_STATUS_ABNORMAL_TERMINATI ON 

The status return value from this IOCTL will be one of the following: 

□ STATUS_SUCCESS 

□ STATUS JtfWALID_DEVICE_REQUEST 

A DUPLICATEREQUEST error is returned if the FSD is already stopped. 
Cache management interfaces 

The following two interfaces are defined for use by the ECM to potentially invalidate 
data in the NT File Cache. 

These IOCTL control codes will be defined for cache management for the eStream FSD: 

IOCTL_EFS_INVALIDATE_FILE 

I OCTL_E FS_ I NVAL I DATE_D I R_CONTENTS 

Invalidating a file 

The input buffer for the INVALID ATE_FILE IOCTL should have the following: 
□ handle: 4-byte EFSDFileHandle for the open file that must be invalidated 
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The output buffer for this IOCTL will be filled with the following: 



□ status: 4-byte value, with one of the following: 



EFS 
EFS 
EFS 
EFS 



STATUS 
STATUS 
STATUS 
STATUS 



SUCCESS 

BUFFER_TOO_SMALL 
FILE_NOT_OPEN 
ABNORMAL TERMINATION 



The status return value from this IOCTL will be one of the following: 

□ STATUS_SUCCESS 

Q STATUSINVALIDDEVICEREQUEST 

If in fact the file is open, but not present in the NT File Cache, this IOCTL will simply 
succeed; no error is returned. 

Invalidating directory contents 

The input buffer for the INVALIDATEDIRCONTENTS IOCTL should have the fol- 
lowing: 

Q handle: 4-byte EFSDFileHandle for the open directory whose contents must be 
invalidated 

The output buffer for this IOCTL will be filled with the following: 

□ status: 4-byte value, with one of the following: 



□ STATUS_SUCCESS 

□ STATUSMVALIDDEVICEREQUEST 

General file system requests 

All file system requests that cannot be completely handled by the EFSD will be passed on 
to the ECM. Since the ECM is likely to be a user-mode service, the EFSD cannot call it 
directly; thus these "calls" are made by having the ECM send lOCTLs to the EFSD to get 
and fulfill requests. Each file system request requires multiple lOCTLs sent from the 
ECM to the EFSD: 



EFS_STATUS_SUCCESS 
EFS_STATUS_BUFFER_TOO_SMALL 
EFS_STATUS_FILE_NOT_OPEN 
EFS_STATUS_ABNORMAL TERMINATION 



The status return value from this IOCTL will be one of the following: 
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1 . The ECM sends an 10CTL to the EFSD to get the next request 

2. The ECM sends a second and/or third 10CTL to finish the request 

The following IOCTL control codes will be defined by the EFSD for use by the ECM: 



IOCTL_ 


_EFS_i 


ioctl] 


_efs_: 


ioctl" 


_ EFS J 


IOCTL 


J3FS_ 


IOCTL 


EFS_ 


IOCTL_ 


~EFS_. 


IOCTL_ 


! efs j 


ioctl] 


]efs_: 


IOCTL_ 


~ EFS J 


IOCTL 


]efs_: 


IOCTL 


~efs_: 


ioctl] 


]efs_: 


IOCTL 


efs ; 



FINISH CLOSE 



For the DeviceloControlO call sending IOCTL EFS GET REQUEST, these parameters 
are invariant: 



□ the 10 control code will be IOCTL EFS GET REQUEST 

□ input buffer pointer will be NULL 

□ input buffer size will be 0 

□ output buffer must be non-NULL 

□ output buffer size must be at least 40 bytes — this is the largest buffer needed for 
any request (subject, of course, to slight modifications) 

□ pointer to bytes returned will be non-NULL 

□ overlapped pointer will be NULL 



The IOCTL EFSJRETRY REQUEST is sent by the ECM (or some other user-space cli- 
ent component) to tell the EFSD that, yes, it needs to delete all intermediate information 
about a request already sent back with a GETREQUEST call, and put the request back 
on the list for the ECM to retrieve. This eases implementation issues for the ECM. The 
input buffer for a RETRY_REQUEST is: 



□ request id of the previously retrieved request 
There is no output buffer for a retry request IOCTL. 



What follows is a list of file system requests from the NT I/O Manager, and (he IOCTL 
calls needed from the ECM to service those requests. For all cases, if the EFSD writes to 
the output buffer for an IOCTL, the "bytes returned" field is written with the number of 
bytes written. 
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Create 

This is used for both create and open, for files and directories. 
GET_REQUEST 

The output buffer for the lOCTL EFS GET REQUEST will be filled with the follow- 
ing: 

□ type: a 4 byte field that indicates a Create request 

□ request id: a 4 byte field that will be subsequently sent in the calls to match this 
request 

□ retry count: 4 bytes — how many retries this GETREQUEST corresponds to. 
First time, this is 0. 

□ flags: 4 bytes, one or more of the following ORed together 

o CREATEJ3NLY: fail if file exists already 

o OPENONLY: fail if file does not exist already 

o TRUNCATE: overwrite existing file 

o DIRECTORY: create a directory 

o FILE: create a plain file 

o DELETE ON CLOSE: delete file on last close 
o 1GNORE CASE: obvious 

□ permissions: 4 bytes, one or more of the following ORed together 

o READ 
o WRITE 
o EXECUTE 

□ length of filename: 4 bytes, specifying the byte size needed for the Unicode string 
sent in the next call 

Total size of output buffer: 24 bytes 

GET_CREA TEJNAME 

The input buffer for this IOCTL should have the following data: 

□ request id: the id sent in the previous call 

The output buffer for this IOCTL will be filled with the following information: 

□ request id for this transaction 

□ fully qualified name as a counted Unicode string (including drive letter, if any): 
the length needed was sent back in the GET REQUEST call 

FINISH_CREATE 

The input buffer for this call should have the input buffer filled as follows: 
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□ request id: the matching id from the GET REQUEST call 

□ status: the NTSTATUS result from this request 

□ handle: the 4-byte handle for this opened file, that can be used for subsequent file 
system requests. A unique value will indicate a bad handle, and a failed Create 

□ a Metadata buffer: the metadata for the created/opened file/directory. 

The output buffer for this IOCTL should be NULL. 

Note that a TRUNCATE Create request should cause the metadata sent back to reflect the 
possibly new (zero) length. 

Close 

This closes a handle of a previously opened file or directory. The EFSD will optionally 
send the updated metadata for this file in the GET REQUEST output buffer. If the file 
has been modified in any way, the metadata fields will be non-zero; else they will all be 
zero. 

GET_REQUEST 

The output buffer for this call will be filled with the following: 

□ type: a 4 byte field that indicates a Close request 

□ request id: 4 bytes, for use in subsequent calls for this request 

□ retry count: 4 bytes— how many retries this GET REQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes, the handle for the previously opened file 

□ metadata for this file/directory: 24 bytes 

o creation time stamp 

o modification time stamp 

o file/directory size in bytes 

o attributes (as described above) 

Total size of output buffer: 40 bytes 

FINISH_CLOSE 

The input buffer for this call should contain the following: 

□ request id for this transaction 

□ status: the NTSTATUS for this request 

Read 

This is used for reading file data. 
GET REQUEST 
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The output buffer for the IOCTLJEFSGETREQUEST will be filled with the follow- 
ing: 

□ type: a 4 byte field (hat indicates a Read request 

□ request id: a 4 byte field that will be subsequently sent in the IOCTL to match this 
request 

□ retry count: 4 bytes— how many retries this GETREQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes, the handle for this previously opened file 

□ offset: 4 bytes, the file offset, in bytes, to read from 

□ length: 4 bytes, the length of the read, in bytes 

NOTE: The buffer requested in the (offset, length) pair will not cross a 4K page bound- 
ary. 

Total size of output buffer: 24 bytes. 
FINISH_READ 

The input buffer for this call should have the input buffer filled as follows: 

□ request id: the matching id from the GET RE QUEST call 

□ status: the NTSTATUS result from this request 

□ the number of bytes successfully read; 0 on error 

□ the data from the read; not present on error 

The output buffer for this IOCTL should be NULL. 
Write 

This is used for writing file data. 
GET__REQUEST 

The output buffer for this will be filled with the following: 

□ type: a 4 byte field that indicates a Write request 

□ request id: a 4 byte field that will be subsequently sent in matching calls for this 
request 

□ retry count: 4 bytes— how many retries this GET REQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes, the handle for this previously opened file 

□ offset: 4 bytes, the file offset, in bytes, to write to 

□ length: 4 bytes, the length of the write, in bytes 

□ file length: 4 bytes, the length the file will be if this write succeeds 
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Total size of output buffer: 28 bytes. 

NOTE: The buffer requested in the (offset, length) pair will not cross a 4K page bound- 
ary. 

GET_WRITE_DATA 

This JOCTL will have an input buffer with: 

□ request id for this transaction 

□ status: if not STATUS_SUCCESS, this ends the request; the output buffer is un- 
touched, and no FINISH_WRITE call is expected. 

And the output buffer will be filled with: 

□ request id 

□ data buffer for the write— the byte length sent in the previous GET REQUEST 
FINISH_WRITE 

For this finishing request IOCTL, the input buffer has these contents: 

Q request id: the matching id from the GETREQUEST call 
o status: the NTSTATUS result from this request 

□ bytes actually written; should be equal to requested bytes unless failure occurs. 
Rename 

This is used for renaming a file or directory. 
GET_REQUEST 

The output buffer for this IOCTL will be filled with the following: 

□ type: a 4 byte field that indicates a Rename request 

D request id: a 4 byte field that will be subsequently sent in the FINISHREQUEST 
call to match this request 

□ retry count: 4 bytes— how many retries this GET REQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes; the handle for this previously opened file or directory 

□ length of target name: 4 bytes; the byte length needed for a counted Unicode 
string for the target name 

Total size of output buffer: 20 bytes. 

GET_RENAME_ TARGET 
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The input buffer for this call will have the following: 

□ request id for this transaction 

□ status: if not STATUSSUCCESS, then this terminates the request: the output 
buffer is not touched, and no FIN1SHRENAME call should be sent 

The output buffer will be filled with the following: 

□ request id 

□ target name: a counted Unicode string, using the same number of bytes as sent in 
the GETREQUEST output buffer 

FINISH^RENAME 

The input buffer for this call should have the following: 

□ request id 

□ status: NTSTATUS for the transaction 
Delete 

This is used for deleting a file or directory. 
GET^REQUEST 

The output buffer for this call will be filled with the following: 

□ type: a 4 byte field that indicates a Delete request 

□ request id: a 4 byte field that will be subsequently sent in the call to match this re- 
quest 

□ retry count: 4 bytes— how many retries this GET REQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes; the handle for this previously opened file or directory 

Total size of output buffer: 16 bytes. 
FINISH^DELETE 

The output buffer for this call should be NULL. 
The input buffer should have the following contents: 

□ request id: matching id from the GET REQUEST call 

□ status: NTSTATUS of this request 
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Metadata read 

This is used for requesting metadata about a file or directory. 
GET_REQUEST 

The output buffer for this call will be filled with the following: 

□ type, a 4 byte field that indicates a Metadata request 

□ request id: a 4 byte field that will be subsequently sent in the call to match this re- 
quest 

□ retry count: 4 bytes— how many retries this GETREQUEST corresponds to 
First time, this is 0. 

□ handle: 4 bytes; the handle for this previously opened file or directory 

Total size of output buffer: 16 bytes. 
FINISH_METADA TA_READ 
The output buffer for this IOCTL will be NULL. 
The input buffer should have the following contents: 

□ request id: id from the corresponding GET REQUEST 

□ status: NTSTATUS for this operation 

□ the following data about the file or directory: 

o creation time stamp 

o modification time stamp 

o file/directory size in bytes 

o attributes (as described above) 

Metadata write 

This is used for setting metadata for a file or directory. 
GET_REQUEST 

The output buffer for this call will be filled with the following: 

□ type: a 4 byte field that indicates a Metadata Write request 

□ request id: a 4 byte field that will be subsequently used for all calls for this request 

□ retry count: 4 bytes— how many retries this GET REQUEST corresponds to 
First time, this is 0. 

□ handle: 4 bytes; the handle for this previously opened file or directory 

□ metadata for this file/directory: 24 bytes 

o creation time stamp 
o modification time stamp 
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o file/directory size in bytes 
o attributes (as described above) 

Total size of output buffer: 40 bytes. 

FINISH_METADA TA_ WRITE 

The input buffer should have the following contents: 

□ request id: from the previous call 

□ status: NTSTATUS for this request 

Component design 

This section is organized in the following manner: 

1 . General layout of the eStream file system driver 

2. Genera] observations about the low level design 

3. Organization of data structures 

4. Description of the algorithms for communication with the ECM 

5. Description of each dispatch routine 

Layout 

The EFSD will be generally organized in the following manner: 
Q All major IRPs will have their own dispatch routine. 

□ All actual I/O requests to the ECM will be generalized from the dispatch routines 
to a set of routines that handle the communication with the ECM, to isolate this 
aspect. 

□ All utility functions will be in their own file or files. 

General points 

The design of the EFSD will look a lot like the sample FSD from Rajeev Nagar's NT FS 
Internals book, which looks a whole lot like the Fastfat FSD source from the NT IFS kit. 

Here is a list of general points that can be made about the EFSD: 

□ Any IRP that can be handled asynchronously will be posted to a work queue; this 
means that the dispatch routine for such an IRP must be able to handle being 
called in a context other than the original requestor. 

□ There are no volumes, and no Volume Parameter Block or Volume Control 
Block. There isn't a VPB for a network redirector; I've verified this with the 
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LanManager redirector. Hence we don't have to support any operations on a 
volume in EFS. 

□ We will not allow the creation of paging files in EFS. There is a bit available for 
a Create IRP that specifies this, and we can complete the IRP with an unimple- 
mented error return code. 

□ All file synchronization will be on a File Control Block (FCB) basis, using the 
standard Resource and PagingloResource ERESOURCE objects used by the rest 
of the Windows Executive. 

o User requests will be synchronized by acquiring the main Resource 

shared for reads, shared for most writes, and exclusive for file size 
changes, deletion, etc. 

o Paging I/O requests will be synchronized by acquiring the PagingloRe- 
source— again, shared for reads, shared for most writes. Exclusive access 
will be needed to set file sizes. 

□ Most disk file systems have a resource associated with a VCB, which is acquired 
exclusively for creation/deletion etc. We will have a global EFS resource for this, 
since there are no VCBs. 

□ Asynchronous requests will be handled by posting the IRP to the Critical- 
WorkQueue, and marking the IRP as pending. 

o A common worker routine will be used for all async posts, which will dis- 
patch the IRP to the appropriate real IRP routine when it's invoked. 

o An async request will be defined as one that IoIsOperationSynchronous() 
returns false, and the EFSD is the top-level component (see below) 

□ The EFSD will track the top-level IRP for the thread whose context it is running 
in. In particular, 

o No async processing request will be honored unless the EFSD is the top- 
level component 

o No cache manager requests will be made unless the EFSD is the top-level 
component 

o EndOfFile size— that is, the true size of the file — will not be extended or 
changed by paging I/O 

□ EFS will not support holes in files, and hence the ValidDataLength FCB field will 
be set to disable this. This means the AllocationSize for an eStream file/directory 
will always be equal to the EOF size. 

□ Most fast I/O routines will be supported in EFS. We will use the FSRTL supplied 
routines for fast reads and writes. 

Q All cache manager resource acquire/release callbacks will be supported. All will 
point to common routines that simply acquire or release the main or paging I/O 
resource for the FCB. The Context pointer passed into all of them will be the 
FCB for the stream. 

□ Synchronous read/write requests will update the CurrentByteOffset in the File ob- 
ject. 

□ Each Create will result in a unique Context Control Block (CCB) data structure; 
this will be small, and only hold those few fields needed: 

o For the Directory Control IRP, a CCB needs to hold the current entry in- 
dex and the pattern originally used — for subsequent queries 
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o A field for various flags 

□ A single FCB will represent all current open instances of a file. When a Create 
request comes in, the EFSD will search the current open FCBs to try to find one 
matching this file/directory name. 

o For now, this will be a hash table on the file name. We can improve this 
as needed. 

o The EFS global resource must be acquired exclusively: 

■ before the global FCB data structure is searched. Why? If it's 
just a read, can't we acquire it non-exclusively? 

■ before a new FCB is added to the list 

■ before an FCB is deleted from the list 

a EFS will not support open by file ID; hence the Filelntemallnformation class for a 
File Information IRP will not be supported. 

□ Actual I/O will be directed to standard routines in a separate file, so they can be 
isolated and updated easily as our method of transferring data changes. 

□ Here's how to do file/directory renames: 

o The I/O manager will send to the EFSD this sequence: 

■ Create for source 

• Create for target, with the SLOPENTARGETDIRECTORY 
flag set 

■ Set Information with a Rename request for the source, sending the 
target directory FileObject handle and the target name in the 
FilelnformationClass record. 

o EFSD needs to do this: 

■ When it receives the Create for the target and the target directory 
exists, return STATUS_SUCCESS, and change the name in the 
FileObject to the basename of the target (the full pathname of the 
target is sent in), and set the Status.lnformation to FILE EXISTS 
or FILE DOES NOT EXIST, as appropriate. If the target directory 
doesn r t even exist, return PATH NOT FOUND. 

■ When it receives the Set Info request, if all the flags check out 
(e.g., if the file exists, ReplaceExisting must be TRUE), send a Re- 
name request to the ECM 

□ Reads and writes to only regular files will be supported, not to directories. 

a Any code that touches user buffers or can call routines that may throw exceptions 
must be guarded by a try/except block. 

□ Some tips on memory allocation (from /perforce-doc-dir/osrdocs/defensive- 
driv.html) 

o We will use our own memory allocation/deallocation routines, instead of 

ExAllocatePoolQ et al. directly 
o These routines can do various checks for trashing memory: 

■ fill allocated memory with a pre-defined bit pattern, instead of ze- 
roes; fill deallocated memory with a different pattern. 

■ allocate a header/trailer with standard information, like where allo- 
cated, from what pool, etc. 
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» change the bit pattern in the header/trailer on deallocation, and 
look for freeing memory twice 
o We probably want to allocate using lookaside lists, since we'll be allocat- 
ing and deallocating smallish chunks of memory for our data structures. 

Data structures 

The following are major data structures used internally by the EFSD. Data structures 
used to communicate with the ECM are described in the following section. 

Nodeldentifier 

A Nodeldentifier is a simple structure that starts all other structures used in the EFSD. 
This makes a good debugging check to insure that we receive and are operating on the 
right type of data. It consists of two fields: an identifier field, and the total structure size. 

FCB 

The FCB is a critical data structure for the file system driver. There is one FCB structure 
allocated for each unique file or directory that is currently open — regardless of how many 
open handles there are for this entry. Multiple CCB structures can point to a single FCB. 

Logically at least, part of an FCB is exposed to the Cache Manager and VMM to support 
caching and paging I/O. We will follow the example of Rajeev Nagar's book, and embed 
the FSRTL COMMON FCB HEADER and other required structures directly in an 
FCB. 

Here are the basic contents of an EFSD FCB: 

□ The required FCB contents above 

□ An open handle count: incremented on Create, decremented on Cleanup 

□ A reference count: incremented on Create, decremented on Close. The semantics 
of NT file requests require these two be used together to determine when an FCB 
can be deleted 

□ A pointer to the next FCB on its hash list 

Q A pointer to the first CCB opened for this FCB 

□ The fully qualified name of the file/directory 

□ The Metadata associated with this file/directory 

□ A SHAREACCESS structure used to check sharing violations 
o Various flag bits 

CCB 

A CCB represents a currently opened handle to a file or directory. If two processes have 
the same file opened, there will be a unique CCB allocated for each process. 
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Here are the basic contents of an EFSD CCB: 

□ A pointer to its corresponding FCB 

□ A pointer to the next CCB opened for its FCB 

□ A pointer to the FileObject opened for this file/directory 

□ The current file index for a directory query 

□ The directory search pattern used for directory queries for this opened handle 

□ Various flag bits needed 

IrpContext 

An IrpContext structure encapsulates the interesting data from an IRP, and the current 
stack location, for easy access. This structure is allocated on entry to dispatch routines, 
and used during processing, before being deallocated on exit. 

Communicatipg with the ECM 

I tend to divide a file system driver into two logical parts: 

1 . A front-end that understands the NT FSD interfaces and semantics 

2. A back-end that actually perform the requested actions 

Our back-end is the (admittedly ugly) interface for communicating with the ECM, which 
currently sits in user-space. It's very important to design the ECM such that its front-end 
and back-end are nicely separated: since the ECM may move to kernel space, or we 
might find better interfaces for them to communicate with each other, we need to design 
with this in mind. 

Given the current ECM interfaces defined above, here is a basic design to handle them: 

a An EfsdRequest object will be created for each request that must be handled by 
the ECM. 

□ Each dispatch routine that results in a request to the ECM will allocate an 
EfsdRequest and send it to a common routine for further processing. 

□ New requests will be placed in a NewRequest queue. 

□ Requests that have been "sent" to the ECM, but not yet finished, will be removed 
from the NewRequest queue and placed in a PendingRequest list. 

□ Finishing a request entails removing it from the PendingRequest list, returning the 
contents to the dispatch routine, and destroying the request object. 

Data structures 
EfsdRequest 

This contains: 
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□ request id: a number that uniquely identifies this request 

□ type: the type of request (e.g., Create, Write) 

□ FCB pointer for the file/directory for this request 

□ IrpContext pointer for this request 

□ a kernel event object used for signaling that the request is satisfied 
NewRequestSemaphore 

The EFSD device object's device extension will contain a semaphore dispatcher object, 
initialized to non-signaled, and with an initial limit of MAX LONG. When a request is 
added to the NewRequest queue, this semaphore is released (and the count is incremented 
by 1); the GETREQUEST IOCTL will wait on this semaphore object (which decre- 
ments the count by 1). 

NewRequestQueue 

This is actually a kernel-managed interlocked list that is allocated in the device extension 
area, guarded by a spin lock that's also located in the device extension. Requests will be 
added to the tail, retrieved from the head. 

PendingRequestList 

This list must be searched when an ECM non-GETJREQUEST IOCTL is received, so we 
can't use an interlocked list. We'll use a global single-linked list structure, with elements 
allocated from a dedicated lookaside list using non-paged memory. Before a thread can 
access the list elements, it must acquire a mutex. 

Algorithm 

□ All dispatch routines, and asynchronous read/write routines, will call LowLevel- 
PostRequest() to have their requests satisfied. LowLevelPostRequest() is itself 
synchronous; that is, the code calling it is something like this: 

status = LowLevelPostRequest (f cb, irp_context) ; 
/ / we're done ; 

// do all deallocation and cleanup needed 
IoCompleteRequest (irp, . . . ) ; 

□ LowLevelPostRequest() will do the following: 

if this is a read or write IrpContext 

see how many requests are needed to satisfy the IRP: can't span page boundary 
allocate the N requests 

allocate an array of N event pointers to hold the request events 
assign the pointers to the array 
if > THREAD J/VAITJDBJECTS 

allocate an array of N PKWAIT_BLOCKS 
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else if this is a directory query IrpContext 
look at the FileSize of the directory FCB 

allocate enough read requests to read all directory data from the ECM 
as above, allocate an array of event pointers, wait objects (if necessary) 

else 

allocate a new EfsdRequest for the incoming FCB and IrpContext 
place all requests on the NewRequestQueue, using ExlnterlockedlnsertTailList() 
call KeReleaseSemaphore() on the NewRequestSemaphore 
if multiple requests generated 

call KeWaitForMultipleObjects() on the request object's event 

else 

do a KeWaitForSingleObject() on the request object's event 
when the event(s) is/are signaled 
fill in the values into the IRP 
deallocate the EfsdRequest(s) 
return status 

□ When a GETJREQUEST IOCTL comes in from the ECM, the EFSD will do the 
following: 

do a KeWaitForSingleObject() on the NewRequestSemaphore 
remove the first request from the NewRequestQueue, 

using ExlnterlockedRemoveHeadList() 
lock the PendingRequestList mutex 
place this request on this list 
release the mutex 

fill in the IGCTL output buffer identifying the request 
complete the IOCTL IRP 

□ When a RETRY^REQUEST IOCTL is received, the following takes place: 

lock the PendingRequestList mutex 
search the list by request id; error if not found 
remove the request from the list 
release the mutex 

enqueue the request on the NewRequestQueue — on the list head 
release the NewRequestSemaphore 
complete the IOCTL IRP 

□ When any "finishing" IOCTL is received — i.e., the second of two or the third of 
three — the following is done: 

acquire the PendingRequestList mutex 

search the list by request id; error if not found 

remove the request from the list 

release the mutex 

do all buffer copying, set all flags, 

and otherwise insure the input IRP has the correct state 
signal the EfsdRequest event 
complete the IOCTL IRP 

□ When any other request IOCTL is received — e.g., the second of three— this is 
done: 
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acquire the PendingRequestList mutex 
search the list by request id; error if not found 
release the mutex 

fill in the output buffer of the IOCTL as appropriate 
complete the IOCTL IRP 

Dispatch routines 
DriverEntry 

This does a whole slew of initialization, including the dispatch table, fast I/O table, the 
cache callbacks, the FCB hash table and its synchronization object, creates the FS device 
object, and sets up the interface with the ECM. 

Create 

There is one Create routine; there will be no async processing of Create requests. Ulti- 
mately, its job is to send a create request to the ECM, and return SUCCESS or not to its 
caller. Here is a general algorithm for this routine. 

create an IrpContext 
if a page file is requested 
return error 

generate the absolute pathname of the requested file: 
if OPEN_TARG ETD I R ECTOR Y specified 
if OPEN_ON L Y not specified 
return error 

generate the pathname of the parent directory of the requested file 
if a related file object is specified 
if the related file is not a directory 

return error 
if the related filename doesn't start with 7 

return error 
if the input filename starts with 7 

return error 

concatenate the input filename with the related file directory 
else use the input filename 

if the input filename does not start with 7 
return error 
acquire the global EFS resource exclusively 
search the FCB hash table for this file by name 
if not found 

call LowLevelPostRequest() to send the request to the ECM 
if error is returned from ECM 

return the correct error: FILE NOT FOUND or PATH NOT FOUND 
create a new FCB and add to the hash table 
call loSetShareAccess() for this FCB 

else 

check input attributes/flags against those in FCB 
if opening for write or delete on close 
call MmFlushlmageSectionO 
if this fails 
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return error 

if failing mismatch — e.g., if loCheckShareAccess() fails 
return error 
create a new CCB for this file 
set all appropriate flags on CCB and/or FCB 

COMMON^FCBJHEADER flag is set to FastlolsPossible 
set all fields on input FileObject: 

write through flag 

FsContext points to common FCB header 
FsContext2 points to CCB 
if OPENTARGETDIRECTORY is specified 
search for the input target object: 

look for this in the FCB hash table 

if not found 

send a Create request for this file to the ECM 
if this target filename exists 
set Status.lnformation to FILE_EXISTS 

else 

set Status. Information to FILE_DOES_NOT_EXIST 
if a Create was sent to the ECM for the input FileObject 
send a Close request for the input target to the ECM 
change the name in the FileObject to the basename of this file 
the CCB and FCB remain opened for the target directory 
all necessary data structures should be deallocated, for success and error 
release the global EFS resource 

Cleanup 

No async posting of Cleanup requests will be done. Algorithm: 

acquire the global EFS resource, and the FCB Resource, for this file, exclusively 
if this file is marked for deletion 

if this is the last open handle for the file 

acquire the PagingloResource exclusively 

set the file size in the FCB to 0 

release the PagingloResource 

purge the cache, if necessary, with MmFlushlmageSection() 
call LowLevelPostRequest() to send a Delete request to the ECM 
decrement the count of open handles in the FCB 
if caching is on 

flush the cache by calling CcUnitializeCacheMaps() 
any time stamps must be updated if accesses were done using fast I/O 
set the FO_CLEANUP_COMPLETE flag in the FileObject 
call loRemoveShareAccess() 
release the global EFS and FCB Resources 

Close 

There will be no async posting of Close requests. Note that we only send a Close request 
to the ECM if this is the last close for an open file — i.e., we're matching Close requests 
with Create requests. 

Algorithm: 
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acquire the global EFS resource exclusively and the FCB Resource 
deallocate the CCB 

decrement the reference count for the FCB 
if the FCB ref count is now 0 

remove the FCB from the hash table 

deallocate the FCB 

call LowLevelPostRequest() to send a Close request to the ECM, 

updating metadata if necessary 
release the global EFS resource and the FCB Resource 

Read 

Reads will definitely be open to async posting. A general algorithm: 

if the read length is 0 

return success 
if the target file object is a directory 

return error 

if this IRP can be handled async 
post for async processing 

if this read is non-buffered, and it's not for paging I/O, and 

there is a mapped data section object for this file 

acquire the FCB main Resource exclusive, and the PagingloResource shared 

call CcCacheFIush() on the range of this read (current byte offset, length) 

release the FCB resources 
if this is for paging I/O 

acquire the PagingloResource shared 

else 

acquire the main Resource shared 
if this read starts beyond EOF 

return EOF 
if the read length goes beyond EOF 

truncate the length 
if this is a buffered read 

if the PrivateCacheMap is NULL 
call CclnitializeCacheMap() 

if this is an MDL read 
call CcReadMdl() 

else 

call CcCopyRead(), using either an allocated MDL or the UserBuffer 

else (it's non-buffered) 

call LowLevelPostRequest() to send a Read request to the ECM 

if this is a synchronous, non-paging read 

update the current byte offset in the FileObject 
set the number of bytes read in the Status. Information field 
release any acquired resource and deallocate appropriate data structures 

Write 

Writes will definitely be open to async posting. A general algorithm: 



Omnishift Technologies, Inc. 



23 



Company Confidential 



eStream File System Driver Low Level Design 



if the write length is 0 

return success 
if the input file is a directory 

return error 
if the file not opened with write permissions 

return error 

if this IRP can be processed asynchronously 
post for async processing 

if this is a buffered write 

call CcCanlWrite() 

if false 

we have a hard error; fail 
if this is paging I/O 

acquire the PagingloResource shared 

else 

acquire the main Resource shared 
if the length is 

(Low == FILE J/VRITEJTO_END_OF_FILE) && (High == Oxffffffff) 
we're to write at EOF 
if this is a non-paging, non-buffered write, and 

there is a mapped data section object for this file 
acquire the global EFS resource exclusive 
acquire the PagingloResource shared, starving exclusive waiters 
call CcCacheFlush() on the range for this write 
release the PagingloResource 
return error if the cache flush failed 

acquire and release the PagingloResource exclusive (to serialize) 
call CcPurgeCacheSection() on the range for this write 
release the global EFS resource 

if this is a paging I/O write 

if the starting offset is beyond the current EOF 

return success 
if the ending offset is beyond the current EOF 

truncate the write length to EOF 

if this is a buffered write 

if the private cache map is NULL 

call CclnitializeCacheMap() for this file 
if the write will extend the file size 

release the resource acquired 

re-acquire the resource exclusive 

call CcSetFileSizes() inform the cache manager 
if this is an MDL write 

call CcPrepareMdlWrite() 

else 

call CcCopyWrile(), using either an allocated MDL or the UserBuffer 

else 

call LowLevelPostRequest() to send the write request to the ECM 

set the number of bytes written in the Status. Information field 
update the file size fields in the FCB if this write extends the length 
if this is a non-paging write 
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set the CCB modification flag 
if this write is synchronous 

update the CurrentByteOffset field in the FileObject 
release any acquired resource and deallocate appropriate data structures 

Fast I/O Read 

Initially at least, well just set the fast I/O read routine to FsRtlCopyRead(). 
Fast I/O Write 

Initially at least, well just set the fast I/O write routine to FsRtlCopyWrite(). 
Fast I/O Query Basic Info 

This will just fill in the basic info buffer with the data in the FCB. 
Fast I/O Query Standard Info 

This will just fill in the standard info buffer with the data in the FCB, 
Fast I/O Query Open 

This will just fill in the network open info buffer with the data in the FCB, if the file ex- 
ists. Some empirical observations I've made using NTFS: 

• Regardless of whether the file exists or not, this will return TRUE (all fast I/O 
routines are boolean) 

• If the file does not exist, it will set the EOF size in the buffer to 0. The Alloca- 
tionSize must be non-zero. All other fields seem to be don't cares. 

• If the file exists but is zero length, then both EOF and AllocationSize will be 0. 

• The IRP sent to this routine is for an IRPJVIJCREATE; we can use more than 
just the name to identify the file, but also the security characteristics or whatever 
else is sent in the IRP. 

File Query Info 

Standard queries will be supported; these however will not: 

• Filelntemallnformation— no OPENJ3Y FILEJD 

• FileEalnformation — no EA data 

• FileCompressionlnformation— no on-disk compression 

• FileStreamlnformation — no multiple streams 

File Set Info 

These actions will be supported: 
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• EndOfFile size changes 

• AllocationSize changes 

• Time stamp changes 

• File position changes 

• File disposition changes — delete pending 

• File rename requests 

Here is a general algorithm: 

if AllocationSize or EOF size change 

if the new size is smaller than the current size 

call MmCanFileBeTruncated(), to ask the VMM if this is okay 
if yes 

call CcSetFileSizes() 

else 

return STATUSJJSER_MAPPED_FILE error 

else 

send a Metadata Set request to the ECM with new, extended size 
if status returned is error 

return error (disk space full) 
update AllocationSize and FileSize fields of required FCB header 

else if time stamp change 

send Metadata Set request to ECM 
if error returned 
return with error 

else if file position change 

update FileObject CurrentByteOffset field 

else if disposition (delete) change 
if the Delete flag in the IRP not set 

clear delete on close FCB flag 
if file already marked for delete on close 

return success 
if file not open with write permission 

return error 
if MmFlushlmageSection() fails 

return error 

if this is a directory, and the directory is not empty 

return error 
set delete on close flag in FCB 

else if rename change 

if the source name is for a directory 

if the directory has any open files/directories under it 
return error 

if Parameters.SetFile. FileObject is NULL (simple rename) 
target dirname is the same as dirname of IRP FileObject 
target basename is Buffer.FileName 

else 

if Buffer.RootDirectory is NULL (fully-qualified rename) 
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fully qualified target name is Buffer.FlleName 

else (relative rename) 

call ObReferenceObjectByHandleQ to get the file object of the relative directory 
from file object, get (fully qualified) dirname of target 

append Buffer.Filename to root directory dirname to get fully qualified target 

if the target name isn't on EFS 

return error 
if target exists 

if Parameters.SetFile.ReplacelfExists is FALSE 

return error 
if target is a directory 

return error 
if target is read-only 

return error 
if target has any open handles to it 

return error 

call LowLeve!PostRequest() to send a Rename request to the ECM 
Directory Query 

Directory queries turn into directory read requests to the ECM. The EFSD will take the 
contents of the read buffers and fill the IRP MN QUERY DIRECTORY buffers sent to 
it from the NT Executive by parsing the directory contents. 

Note: this design does not use the NT Cache Manager for metadata or for directory en- 
tries, nor does the EFSD store the directory contents anywhere in its data structures. It 
will always go back to the ECM for directory queries. Given that most directory queries 
occur in this order: 

Create 

Directory query 
Close 

unless we can cache the contents in a location more persistent than an FCB, we will need 
to resubmit the request to the ECM for each new directory query. If this poses a per- 
formance problem, we need to handle it then. 

Directory queries are subject to async processing. 

This is an ugly NT interface. Here are some general points regarding directory queries, 
and how they will be implemented on EFS: 

• These requests come in from the I/O Manager in a context-sensitive sequence. 
I.e., a request will come for the initial N directory entries; the next request will be 
for the next M entries; etc. Kind of like strtok(). 

• Thus, state must be maintained from request to request. This state will be kept in 
the CCB for a file stream, and consists of: 
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o Pattern sent in on first request 

o Index of n'th entry to start retrieving with 

• My experience is that the INDEXSPECIFIED flag is never set in a directory 
control query, even on queries subsequent to the initial one. 

• For EFSD, the Filelndex will represent the offset of the fixed-length portion of a 
directory entry as returned by the ECM. 

• Initially, at least, all directory queries will cause the EFSD to read all the entries 
for that directory from the ECM. We can modify this later if needed. 

Here is a general algorithm, based on the sources for the Fastfat driver: 

if the FileObject is not for a directory 
return error 

post this for async handling, to read all directory pages from the ECM 

if the CCB pattern field is empty and the CCB flag "match air isn't set 
this is the initial query 
acquire the FCB Resource exclusive 

else 

acquire the FCB Resource shared 

get a pointer to the input buffer, using either an MDL or the UserBuffer 
if this is the initial query 

parse the FileName pattern 

save the pattern in the CCB 

if the pattern is 

set the "match all" flag in the CCB 

if SLRESTARTSCAN is specified 

use an index of 0 for the query 
else if SLJNDEX^SPECIFIED is specified^ 

use the input index for the query 

else 

if this is the initial query 
use an index of 0 

else 

use the index saved in the CCB 
start with the directory entry corresponding to the starting index 

if this index is beyond the number of entries in the directory, and this is not the initial query 

return STATUS_NO_MORE_FILES 
while there are more directory entries 

if the directory entry name matches the pattern in the CCB, 

using FsRtllsNamelnExpression() 
if there isn't room in the buffer for this entry 
break 

write the entry in the input buffer 

if there is a next directory entry beyond the current one 

the Filelndex field is set to the offset of this next directory entry 

else 

the Filelndex field is set to 0 
if there is a previously written entry 

fix up the NextEntryOffset in the previous entry to the byte offset of this entry 
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if SLRETURNSINGLEENTRY specified 
break 

8-byte align the current pointer in the user buffer 
advance to next directory entry 

if we wrote nothing 

if we stopped because of no room 

return STATUS_BUFFER_OVERFLOW 

else 

return STATUS_NO_SUCH_FILE 

update the index field in the CCB 

Status. Information is set to the number of bytes written 

return STATUS_SUCCESS 

File System Query Info 

Empirically, I've noticed that the LanMan redirector returns failure for most of these re- 
quests. So, except for any user-defined FSCTL requests we want to define, I r m going to 
fail all of these until it turns out we need to do otherwise. 

File System Set Info 

Ditto for this IRP type too. 

Volume Query Info 

We at least need to minimally implement these requests: 

• FileFsAttributelnformation 

• FileFsVolumelnformation 

• FileFsDevicelnformation 

This will be handled solely by the EFSD; the request will not go out to the ECM. File 
system size requests will not be handled. 

Volume Set Info 

We will fail all requests of this type. 
Flush Buffers 

A buffer flush request for a file stream will mean the following: 

• If the file stream isn't buffered, return immediately 

• The FCB main Resource is acquired exclusive 

• The Cache Manager is told to flush the buffer for the byte range of the file 

• The resource is released 
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A buffer flush for a directory is a successful NOP. 

The buffer flush request will insure that contents in the NT File Cache are written to the 
ECM (as a normal write request); the buffer flush request itself will not be propagated to 
theECM. 

Testing design 
Unit testing plans 

The EFSD will be tested apart from integrating with the ECM or the rest of the eStream 
client. Some points: 

□ There will be a (relatively) simple stand-in user process for the ECM, to get re- 
quests from the EFSD and handle them locally. 

□ As much EFSD functionality as possible will be done using user-mode test cases 
(e.g., open files, read/write files, delete files, etc.). 

□ Some functionality may need to be unit tested using another kernel-mode driver to 
send explicit IRPs to the EFSD. 

□ Filemon will be used to monitor the requests being handled by the EFSD. 

Stress testing plans 

I've heard of a file system filter driver test package available from Microsoft. This is 
probably the best way we have of stress testing the EFSD. 

Coverage testing plans 

We'll try to measure coverage on the EFSD. If there is a general kernel-mode method for 
measuring coverage that's used company-wide, we'll exploit it. Otherwise, there will be 
some primitive self-coverage instrumentation conditionally inserted in the driver code 
that we can use at least for major code paths. 

Cross-component testing plans 

There's not much to do here apart from normal interactions with the ECM. 

Upgrading/Supportability/Deployment design 

For supportability, there will be solid debugging aids— using printfs— built into the 
EFSD sources. Additionally, aside from good error codes returned from its interfaces, 
the EFSD will explore diagnostic traces optionally dumped for deployment. 
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Issues with stakes in the ground 

□ At present, I am assuming that there is a single drive letter associated with the 
EFSD, though there's no technical reason why this must be so. If indeed we or- 
ganize the client system and the eStream file hierarchies to have multiple drive 
letters, either the EFSD or the ECM will need to parse the drive letter and do the 
right thing. 

□ This design assumes that 8.3 DOS-style filenames need not be supported. Adding 
such support will increase the complexity of the EFSD, as well as many other 
components in the eStream system: on the client, on various servers, and on the 
content builder. 

□ No support is provided in this design for: 

a. byte locks 

b. directory notification 

c. file open by file id 

d. file system control requests 

Open Issues 

1 . I'm unclear on how to use the NT File Cache for metadata and directory contents. 
For now, I'm ignoring such matters, and we will only be caching file contents. 

2. I do not know how to hook up the EFSD to UNC names. That is, I don't know 
how to set things up to have all file accesses like \\ASPl\Office\winword.exe be 
directed to the EFSD by the NT I/O Manager. 

3. This design doesn't cover exactly how ERPs are posted for asynchronous process- 
ing. The SFSD example in Rajeev Nagar's book really isn't sufficient for some 
of what we need to do. Also, it's unclear to me what value there is to returning 
STATUSJPENDING and handling a request asynchronously if the caller is block- 
ing anyway. 



Omnishift Technologies, Inc. 



31 



Company Confidential 



Estream 1 .0 Functional Specification * Use Cases * Revision 0.4 * 



EStream Client Functionality: 

■=> Installation of eStream client code 

o Use browser to contact ASP Web Server, download bits to be installed. 

o Install z: file system hooks & setup to have z: mounted at system boot. 

o Install eStream client code, which services z: file sys requests from local 
cache or from servers & which handles sideband communication w/ 
servers, and setup to activate estream client code at system boot. 

o Install NoCluster.sys to disable page fault clustering at system boot. 

o Install eStream browser plug-in, which can receive messages from ASP 
Per-User Account Server alerting eStream client when new app purchased. 
[Sending unsolicited messages may not be possible thru firewall.] 
■=> Execution of eStream client code 

o Respond to z: file sys requests. For apps w/ active online connection(s), 
user sees the detailed contents on z: that one would see if one had installed 
the apps locally, though copy access may be controlled. For apps to which 
the user has obtained offline access, user also sees the appropriate detailed 
contents on z: (although the files are actually in eStream client-managed 
memory on local disk). For each app whose connection is currently 
inactive, user sees a placeholder file entry on the z: file system (on which 
the user can double-click to launch an active connection). 

o Establish/terminate session logins to ASP Per-User Account Server, upon 
user request or upon receiving app purchase message from browser plugin. 

o Obtain/cache unique certificates for purchased applications from ASP Per- 
User Account Server. 
<=> Register with ASP 

o Use browser to contact ASP Web Server. 

o Follow ASP process to register. 

o User obtains login/password, used by estream client code for sessions, 
o ASP records user's login/password on ASP Per-User Account Server. 
■=> Purchase of application 

o Use browser to contact ASP Web Server. 

o Follow ASP process to buy app; user is given unique certificate for app. 

o App purchase & certificate recorded on ASP Per-User Account Server. 

o User is directed to go to client & request app installation and/or ASP Per- 
User Account Server attempts to send message to eStream browser plug-in 
on user's preferred client system (if any), so client can begin app install. 
*=> Installation of application 

o Send unique certificate for application to appropriate ASP DRM Server, 
get back id for closest/best App Server & a session id. 

o Contact designated App Server using id info, download meta-data about 
app, potentially including registry/DLL/filesys spoofing info, prefetching 
info, initial cache contents for app. For offline installation, obtain all files. 

o Perform initial installation & setup for app, after checking system for 
previously installed version of app & issuing any appropriate warnings. 



<=> Execution of application . 

o Send unique certificate for application to appropriate ASP DRM Server, 

get back id for closest/best App Server & session id. 
o Contact designated App Server using id info, request file system data as 

necessary. Respond to running application's requests, collect usage data. 

Cache portions of application, file system info, & user preference info, 
o Detect server connection issues (apparent loss of connection or connection 

response below acceptable threshold); negotiate with ASP DRM Server 

for alternative connection if need be. 
o At exit from application (or at other selected times), save portions of cache 

to client nonvolatile memory. Upload usage information to ASP Per-User 

Account Server. 
■=> Uninstallation of application 

o Remove all registry/DLL/filesys changes associated with app installation, 
o Remove all meta-data about app. 
*=> Uninstallation of eStream client code 

o Remove z: file system hooks, eStream client code, & nocluster.sys. 

EStream Server Functionality in terms of kinds of eStream Servers responding to Clients 
[may be embodied in any number of physical server computer systems]: 

1 . App server 

o functionally read-only 

o serves .exes, .dlls, etc. 

o contains install info (aka, eStream sets) 

2. ASP web server 

o used to get eStream client bits 
o eStream browser plug-in 

o handle other user queries, e.g., concerning available apps, current billing status 

3 . Per-user account server 

o registration info, issue serial numbers for purchased apps 
o accept/store uploaded info about app usage 
o perhaps: user preferences for each app 

4. DRM server 

o authentication of users 

o validate app licenses, track outstanding offline licenses 
o hands out licenses for #1 above 



Estream Server Management/Maintenance Functionality 
<=> Install/maintain eStream apps [aka Builder] 

o Provide tool/methods to generate initial meta-data about app, including 
registry/DLL/flle spoofing info, initial prefetching info, initial cache 
contents, etc. 

o Provide tool/methods to place app & meta-data into public access area and 

to remove from public access areas 
o Update meta-data as appropriate to reflect uploaded client usage info 
"=> Handle server traffic 

o Support trouble-shooting of performance or correctness problems 



o Perform automated load balancing 
o Support online addition/reconfiguration of servers 
=> Provide tools to process uploaded app usage info. 



en functionality questions: 

«=> Supporting time-based charge for app-usage (e.g., rent by minutes of usage) 
complicates the design & may engender customer support/satisfaction issues. Do 
ASPs want/need this support? [Prefer to steer them away from this model.] 

«=> How should we handle minor upgrades/patches of apps (i.e., service packs)? One 
method to allow active use of previous versions plus availability of new versions 
without treating new versions as if they were entirely new applications would be 
file versioning. 



EStream 1 .0 Top Level Component Breakdown * Revision 0.1 * m BB^ 
Client system components 

•=> Z: File system manager [1] 

o Handles all z: file system requests generated on client 
o Makes requests to EStream cache manager 
o Attempts to filter references that suggest software piracy activity 
■=> EStream client core 

o Session manager [ 1 2] 

■ Handles establishing/terminating ASP sessions 

■ Negotiates for app license & security using user unique certificate 

■ Invoked either by eStream client user interface or by cache mgr 
o Cache manager [2] 

■ Responds to Z: file system manager requests 

■ Maintains client cache of app & file system data/metadata 

■ Requests info as necessary from Estream client networking 

■ Requests session/license for non-mounted apps from session mgr 

■ Consumes/gathers profiling/feedback data 
o File manager [3] 

■ Provides interface to all eStream created/maintained client files 

■ Gets requests from cache mgr, session mgr, file mgr/spoofer, 
registry mgr/spoofer, app install/deinstall, client install/deinstall 

■=> Estream client network interface [8] 

o Handles requests from EStream cache manager 
o Handles protocol interface to/from server 

o Performs compression/decompression, encryption/decryption of packets 

o Detects network problems & reports to session manager for renegotiation 
*=> EStream client user interface [5] 

o Displays error/info messages from any part of eStream code to user 

o Solicits/obtains info (e.g., login/password, app license) from user 
EStream file system manager/spoofer [6] 

o Filters all non-z: file sys requests, redirects non-z: file refs as appropriate 

o Supports operation of eStreamed apps 

o Avoids eStreamed apps interfering with non-eStreamed apps 
■=> Estream registry manager/spoofer [7] 

o Filters all registry refs, handles registry contents for/about eStreamed apps 
■=> EStream application installer/deinstaller [14] 

o Obtains app spoofing/registry/prefs info & initial cache/profile data 

o Prepares system to be able to run app on user request 

o Supports deinstallation of app 
O EStream client code installer/deinstaller [ 1 3] 

o Installs all client Estream code components 

o Supports deinstallation of all eStream components 
«=> NoCluster.sys [4] 

o Disables page fault clustering in the kernel 
Estream browser plugin 

o Optional EStream component which fields unsolicited server messages 



eStream High-Level Design Diagram 
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eStream File System Straw Man Proposal 

Version 0.5 




Purpose 

The purpose of this document is to present a concrete proposal for the functioning of the 
eStream file system. In many places, 1 make some sweeping generalizations about how 
things should work without describing the data structures and interfaces involved in 
implementing them. This document should eventually involve into a design 
specification. 

Issues Not Covered 

This document does not attempt to cover all issues present in designing the eStream 1.0 
product. In particular, the overall authentication/licensing/security architecture is not 
covered in detail here. It is expected that the security functionality will be mostly 
orthogonal to the design of the basic file system functionality. 

Background 

There are a number of different networked file systems out there. Many of them share 
some requirements with eStream. For example, AFS performs client-side on-disk 
caching, while Coda handles serious server redundancy and disconnected operation. 
Personally, I believe that AFS and Coda are the file systems whose designs are most 
relevant to us. For those interested in further background reading, you might also want to 
look at papers covering NFS, CEFS, xFS, DFS, and Zebra. 

Single File System Name Space 

Many modern distributed file systems present the network file system as a single tree 
mounted at some location on the client system, regardless of which server hosts the data. 
(In fact, with AFS, every file on every server in the world can be accessed through a path 
starting with /afs on the client, assuming the client can reach that server and has sufficient 
privileges to do so.) Compared with systems like NFS and Windows sharing, where each 
share is mounted in a different location on the client, the single name space provides 
greater ease of use. 

The eStream file system would present one universal logical file system. Regardless of 
which ASP provider supplies a particular volume, that volume will always be referenced 
via the same path on the eStream file system. That this is desirable or even feasible is 
predicated on the assumption that OTI is the only entity providing all eStream sets. Each 
volume must get a unique identifier and a unique location to be mounted in the file 
system hierarchy. If two different ASPs privide the same volume DD, then the contents of 
those volumes must be identical. This way, we don't have to tag things in the cache 
based on what ASP they came from, and the cache manager doesn't need to know 
anything about ASPs. If done correctly, only the client networking component and the 
LSM need to know about ASPs. 

Volumes 
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A volume is a complete subtree of a file system. Volumes may contain files and 
directories. Volumes may not be mounted in other volumes. A volume is a logical 
grouping of files within the file system and is the unit of replication across servers. An 
application will reside in a single volume. Two applications will never share a volume. 

Volumes are uniquely identified by a 32-bit volume identifier. Each volume additionally 
has an 8-bit version number. This version number is incremented each time any file 
within the volume changes. (See supporting upgrades, below). Note that the volume id 
is globally unique. If two ASPs provide volumes with the same volume number (and 
version), they have identical contents. 

A volume may be replicated on any number of servers. Each SLM server contains a map 
describing the application servers that currently provide each volume. This global 
replication of this table is acceptable because volumes are added or moved infrequently. 

Identifying Files 

Files and directories are uniquely identified by the pair (volume id, file number). This 
tuple is called a file id. Volume id and file number are each 32-bit signed integers. 
Negative values for both volume id and file number are reserved for special purposes, 
leaving us with 2 A 31 possible volume IDs and 2 A 31 possible files per volume. 

Finding an Application Server for a Volume 

The SLM will tell the client which application servers currently provide each volume. It 
may be necessary for the client to periodically poll the SLM to get up-to-date information 
about the state of the application servers. The License and Subscription Manager on the 
client will keep track of the currently subscribed applications and the application servers 
for each of these applications. 

Directories 

Directories are specially formatted files that are used in a special way by the file system. 
They are identified by file ids, just like other files. From a client-server point of view, 
they are read by the client in the same way as other files. Directories contain arrays of 
entries with the following format: 
( volume number, file number, flags, length, filename ) 

The volume number and file number are 32-bit signed integers. The flags are 32-bits of 
flags. The length is 16 bits and is the length of the filename in bytes. The filename is a 
non-NUL terminated Unicode string. The structure is padded with enough Unicode NUL 
characters to make the structure a multiple of 32 bits long. The next directory entry 
begins on the next 32-bit boundary. 

The access token is not part of the directory, as a single access token is required to access 
all files in a particular volume. 
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The volume number is required so that the the client can construct a local directory for 
the root of the directory structure in the same format as other directories (see filename 
parsing below). It also helps to provide a sanity check. 

Accessing Files 

Assuming that a client has a file-id for a file that it wishes to access, the following client- 
server actions must be supported: 

For stat-like information on the file, we need a GetFileMetadata() interface. The client 
would provide the file id it is interested in and the proper access token for this file. The 
server will either return the metadata for the file or an error condition (like access token 
expired or incorrect access token.) The metadata contains the standard Windows 
metadata information, including file length and file access times. 

On a file open (CreateFile in Windows terminology), we need to verify that we have 
access to the requested file. This is probably best accomplished by calling 
GetFileMetadata and verifying that we can get the metadata. This way, we can fail file 
opens gracefully if we don't have an access token. 

On reads (and writes, when we support them), the client will send the file id and the 
access token to the server along with an offset and a length for the read and write. The 
server will respond with the data. Note that the same mechanism will be used for reading 
both files and directories. 

Pseudodirectories 

For those parts of the eStream file system name space that do not belong to any volume 
(such as the root of the file system), the client must construct appropriate directories 
based on the currently installed applications. This is to support filename parsing starting 
at the root of the directory. For example, if the client has word installed with a root of 
/Worddir and it is volume number 3 and Photoshop installed with a root of /Photodir and 
is volume number 4, the client would construct a directory for the root of the entire file 
system containing 

File name, Volume number, file number 
"Worddir", 3, 0 
"Photodir", 4, 0 

(The file numbers are both zero here because 0 is the index of the root directory of each 
volume, and these are the mount points for each volume.) 

When new applications are installed, the root of the file system would have to be updated 
to reflect the newly installed apps. 

Filename Parsing 

Filename parsing is handled one element at a time, starting at the root of the file system. 
Parsing one path name element involves reading the parent directory's contents (from the 
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cache or the app server), searching it for the file matching the next path element's name, 
and getting the appropriate file id so it can do further lookup. 

Volume Versioning... Without File Versioning 

We can provide volume versioning and incremental volume updates without versioning 
each file in the file system. When a new volume is to be provided, we can append any 
new or changed files as new files in the volume, with new volume IDs that weren't 
already present. If a directory's contents have changed, then a new version of this 
directory will be built, with a new file number. This process will proceed from the leaves 
all the way to the root of the file system, eventually resulting in a new root. The old 
versions of things would still be available for old clients to access, but clients wishing to 
access the new version will simply start at the new root, and would thereby get to a 
consistent picture of the volume. Any file or directory that has not changed from the old 
version to the new one need not be replicated, and will be referenced by its old file 
number. (I.e. newly reconstructed directories will contain the old file number for any 
files that haven't changed.) 

If we reserve the first 256 file ids for the root directory, then the version number can be 
the same as the file number for the root directory. 

Note that if we decide that the complexity of this approach is too high, this does not 
preclude always creating a new volume from scratch for each update. 

Constructing File IDs 

It is the job of the builder to produce the volume file to file id mapping and to construct 
all of the directories. Because directories are files identified by file id, this process must 
begin at the leaves of the volume and proceed to the root. 

Note that constructing a new changed volume will consist of finding the diffs between the 
two volumes and producing some new directories. Changed or newly added files will get 
new file numbers, leaving the old ones around. Note that any directory that has had any 
descendents changed must be reconstructed with the new file numbers, and the new 
directory will get a new file number. This process will proceed to the root of the volume, 
which will receive a new file number. 

Server Failover 

All app servers for a particular volume must share the same mapping of file ids to file, so 
server failover is trivial. There might be a performance impact if the new app server 
doesn't have the requrested file in memory. 

Writing Files into the Application Install Directories 

Two approaches have been discussed for the problem of applications that want to write 
files to their install directories. First, this can be handled wholly inside of the eStream 
file system. The cache manager could allow writes to files handled by the eFS, but these 
writes would not be written back to the server. Instead, they would simply be written to 
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the eFS cache and marked non-purgeable. This approach's primary advantage is that it 
does not rely on a file spoofer. 

The other approach is to use the file spoofer to spoof some accesses to the z: drive. Any 
open for read/write access would cause the existing file (if any) to be copied to a location 
on the c: drive, and the file spoofer would then redirect the open to the newly created file. 
The file spoofer would have to keep track of any file created via this copy-on-write 
mechanism and redirect all future accesses to the copy. There are some issues to this 
approach. For example, it is extremely wasteful when files on the z: drive are opened for 
read/write access but are never actually written. However, it does help reduce the 
complexity of the eFS cache, and is trivial to implement if we have to do c: to z: file 
spoofing anyway. 

In either case, to support the creation of new files in an application's install directory, it 
must be possible to modify the contents of directories in the cache. 

If we don't use the file spoofing approach, there is the issue of how we support written 
files when we move to a newer version of a volume. It would probably be necessary to 
walk the cache and make sure that each written file gets placed in the appropriate place in 
the new volume version. This is likely to be non-trivial, because we need to have fiill 
information about the location of each modified file in the file system tree, and would 
need to download enough of the new volume directory structure to place these modified 
files there. 

64-Bit File Access? 

One question we should answer is whether we will support file sizes greater than 2 GB on 
the eStream file system. I'm inclined to say that such support isn't a requirement for the 
1 .0 product, but I also think that the implementation and verification complexity of 64-bit 
file access on the file system is low enough that we might want to consider building it in 
anyway. 

Simplifications 

We could preclude the possibility of an application consisting of more than one volume. 
Future Possibilities 

Epicon seems to make a big selling point of their technology involving "self-healing" of 
damaged application files. Such support could be provided by computing checksums on 
files in the cache. Whether or not we want to support this is an open question. My 
feeling is that it's something we should leave out of 1 .0. 

Outstanding Issues 

Cache organization has not been addressed. 

Finding and downloading the app install block has not been addressed. 
Security in a multiuser system has not been addressed. 
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eStream File Spoofer Low Level Design 



Curt Wohlgemuth 
Version 2.0 



Functionality 

The eStream file spoofer is a kernel-mode driver responsible for redirecting file accesses 
from local file systems to the eStream file system driver. It is implemented as a file sys- 
tem filter driver that traps all IRP requests to the file system device handling drives that 
must be spoofed, and redirects these requests to the EFSD. 

Data type definitions 

The file spoofer will understand entries in the "file spoof database" as they have been 
identified by the eStream builder and installed by the app install manager, but these are 
not defined by this component. 

Entries in this spoof database will have the following entries: 

□ original (fully qualified) path name of file: this resides somewhere on a local disk 
of the client machine 

□ new (fully qualified) path name of the file to spoof to: this resides on the eStream 
file system drive 

The spoof database will reside in the registry, so it can be persistent across reboots, and 
so the file spoofer need not open a file to load it. As applications are installed on a client 
machine, the Application Install Manager will load new spoof entries into the registry, 
and inform the file spoofer that it must reload this database. Similarly, when an app is 
uninstalled, the AIM will remove spoof entries from the registry, and inform the file 
spoofer. 

This proposes that the each spoof entry is a separate name/value entry under a single key 
in the registry: 

□ Name: the original filename 

□ Value: the new filename 

Interface definitions 

The eStream file spoofer is called by two components: 

1 . The eStream client start service, which will start and stop the file spoofer 
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2. The AIM, which will inform the file spoofer to reload the spoof database from the 
registry 

The interfaces called by both of these user-mode components will be in the form of De- 
viceIoControl() calls. The following lOCTL cgdes will be defined for use by callers of 
the file spoofer: 

IOCTL_FS_START_SPOOFING 
I OCTL_FS_STO P_S POOF I NG 
IOCTL_FS_RELOAD_SPOOFJDB 

Starting spooling 

The input buffer for this IOCTL should supply the name of the registry key containing 
the spoof entries as values. The output buffer for this IOCTL is ignored and should be 
NULL. 

This will return either STATUS_SUCCESS, or an error return status if something goes 
wrong. It causes the file spoofer to read the spoof registry entries, and load up each entry 
into memory. 

Note that starting spoofing is currently identical to reloading the spoof database. 
This is called by the eStream client start service on startup. 

Stopping spoofing 

The input and output buffers for this IOCTL are ignored and should be NULL. This will 
return either STATUS_SUCCESS, or an error return status if something goes wrong. It 
causes the file spoofer to clear its memory image of the spoof database. 

This is called by the eStream client start service on shutdown. 

Reload spoof database 

The input buffer for this IOCTL should supply the name of the registry key containing 
the spoof entries as values. The output buffer for this IOCTL is ignored and should be 
NULL. 

This will return either STATUS SUCCESS, or an error return status if something goes 
wrong. It causes the file spoofer to read the spoof registry entries, and load up each entry 
into memory. 

Note that this is currently identical to starting the spoof database. 
This is called by the AIM when a new eStream app is installed. 
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Component design 

The file spoofer will have these major tasks: 

□ Track the following data: 

o all current valid file spoof entries 
o spoof entries by filtered file system 

□ Filter native file system drivers for local drives, intercept all IRPMJCREATE 
and FASTJO_QUERY_OPEN requests, and for spoofed files, change the file- 
name of the FileObject associated with these requests. 

Data structures 

The file spoofer needs to be able to quickly look up a filename in the in-mernory spoof 
database. The current design will use a hash table, whose size and hash function will be 
tuned as we get experience with real applications. 

Adding or deleting entries from the hash table will by synchronized using a global re- 
source. 

Algorithms 

Here are basic algorithms for these steps. 
Load spoof database 

This reads all the name/value pairs under the registry key which holds the spoof entries, 
loads them into a temporary hash table, then points the real hash table to this one. 

traverse the registry until the input registry key is opened, using ZwOpenKey() 
if not found 

return error 
if no name/value pairs exist in this key 

return "no data" 

for each name/value pair found with ZwEnumerateValueKey() 
build a hash node for this and insert it into temp hash table 
if the drive letter for the old filename entry is one we're not currently filtering 
put this drive letter on new drive list 

acquire the hash table resource exclusively 

point the global hash table head pointer to the temp hash table 

release the resource 

for each drive letter on the new drive list 
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look up FS device for this drive 
if we really aren't attached to it 

attach self to this FS device as filter driver 

free the old hash table 
free the drive list 
return success 

Stop spoofing 

acquire hash table resource exclusively 
free the global hash table 
detach self from all filtered FS devices 
release hash table resource 

Trap Create and QueryOpen requests 

acquire the hash table resource shared 
if hash table head pointer is non-NULL 

look up input filename in hash table 
release hash table resource 
if filename not found in hash table 

send I/O request to original target FS driver 

else 

free memory of existing file name in input FileObject 

allocate memory for new, spoofed filename, copy into this memory 

send I/O request to eStream file system driver 

Testing design 
Unit testing plans 

The file spoofer will be tested as a standalone component, apart from the rest of the 
eStream client.. A driver test program will be written to test all functionality and corner 
cases. This includes filtering all FSDs active for a client system, and multiple drives 
handled by a single FSD. 

Stress testing plans 

The file spoofer should be able to work, with little or no performance cost to the system 
as a whole, even when the attached FSDs are under heavy load. Some stress testing will 
be done in this fashion. 
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Coverage testing plans 

If we come up with a method for measuring coverage for kernel-mode components, we'll 
do so for the file spoofer as well. 

Cross-component testing plans 

Not clear if anything need be done here outside of the standard execution of the eStream 
client. 

Upgrading/Supportability/Deployment design 

I don't see any upgrade/compatibility issues for the file spoofer. For support ability, there 
will be a good debugging strategy and sufficient error message return codes for the caller. 

Open Issues 

This is a list of issues that need to be further investigated or revisited during implementa- 
tion. 

1 . We will need to experiment with the hash table to tune it for fast lookup. It's pos- 
sible that we may need to replace the hash table with a faster lookup algorithm. 
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SCENARIO: Install a subscribed application 

LEGEND: use of specialized fonts in material below: 
Bold indicates block or entire scenario 

Bold italic indicates argument or return interface between blocks or scenarios 
Italic designates meta-info about scenario 

BACKGROUND: ways of invoking this scenario 

"=> User subscribes to a new application or requests upgrade of existing app. 

=> User starts using a new client 

*=> Client learns that a new application is available. 

DISCUSSION: Nature of the AppInstallManager 

The AppInstallManager block may not be a generic module installed as part of the 
client installation of eStream; instead, it may be a unique executable associated with each 
particular application. Either way, it is expected that there is certain basic functionality 
associated with this code; that functionality is described below. 

DISCUSSION: Need for checking license at install time 

The scenario below describes that the application's license is checked at install time. 
This may need to be discussed further. Two reasons for license check at install time are: 

(1) may be required by the software vendor's licensing model, and 

(2) allows eStream client to record which ASP subscription to use when app invoked. 

AppInstallManager: invoked vs/DRM server name & application serial number 

ACTION: Establish connection, get license. 

o Perform CheckLicense, which involves asking ClientNetworklnterface 
to send check-license message including application serial number to 
DRM server name (securely) & getting back response, 

■ If response = License ok, then App Server name & session id 
returned, along with application nickname and upgrade flag. If 
upgrade flag set, ask ClientUI to display message advising user 
about upgrade & to solicit response concerning whether user wants 
to continue with current version. If response affirmative or if 
upgrade flag not set, continue with next ACTION. Else, return 
status to AppInstallManager caller. 

■ If response = License not ok y then reason not ok returned. Some 
possible reasons license may not be ok include: 

• app license expired. Ask ClientUI to display message 
advising user about license status & suggesting that user go 
to the ASP web server & renew license. Return status to 
AppInstallManager caller. 

• app not accessible (DRM server did not respond, DRM 
server indicated that application no longer supported, etc). 
Ask ClientUI to display message advising user that app not 



available & suggesting that user go to the ASP web server 
for more info. Return status to AppInstallManager caller. 

«=> ACTION: Get application installation information. 

o Perform GetAppInstallBIock, which involves asking 

ClientNetworklnterface to send get-install-info message including 
session id to App Server name (securely) & getting back response. 

■ If response = success, then pointer to allocated AppInstallBlock 
data is also provided. Control continues with next ACTION. 

9 If response = failure, then status returned to AppInstallManager 
caller. 

o Please note that if the AppInstallManager is an application-specific 
program, it may not request the entire AppInstallBlock contents at once. 

=> ACTION: Check if application already installed. 

o Perform ChecklfAppAIreadylnstalled, which involves using relevant 
info supplied as part of AppInstallBlock to check for the presence of 
certain registry entries and/or files. This is expected to detect both 
previous estream & non-estream installations in effect. If app not already 
installed, continue with next ACTION. If app already installed, ask 
ClientUI to display message advising user about this & to solicit response 
indicating whether to continue with current installation. If response 
negative, then return status to AppInstallManager caller. If response 
affirmative, continue with next ACTION. 

ACTION: Parse AppInstallBlock info, Set up client to make app ready to run 
o A CTION: Handle non-Z: file association information in AppInstallBlock. 
» Pass ptr to non-Z: file association data in AppInstallBlock to 
InitializeFileSpooferData, which checks client's system against 
relevant files (creating any non-Z: files that need to exist initially) 
and which passes that ptr along with filespoofer-data designation 
and application nickname to eStreamClientFileMgr for storage 
in appropriate area on client, 
o ACTION: Handle registry information in AppInstallBlock. 

■ Pass ptr to registry data in AppInstallBlock to 
InitializeRegistrySpooferData, which modifies client's registry 
as appropriate and which passes that ptr along with registry-data 
designation and application nickname to eStreamClientFileMgr 
for storage in appropriate area on client. 

o ACTION: Handle application data in AppInstallBlock. 

■ Pass cache-data designation, application nickname, & ptr to 
initial application cache data in AppInstallBlock to 
eStreamClientFileMgr for storage in appropriate area on client. 

• Pass profile-data designation, application nickname, & ptr to 
initial profile data in AppInstallBlock to eStreamClientFileMgr 
for storage in appropriate area on client. 



■ For any application files to be preinstalled, pass file-data 
designation, application nickname, application filename, & ptr to 
file data in ApplnstallBlock to eStreamClientFileMgr for storage 
in appropriate area on client. 

o ACTION: Perform other application-specific activities as desired. 

■ If the AppInstallManager is generic code, then there is an 
interface to download an extra executable to do additional 
activities. If the AppInstallManager is code specific to the 
application, optional extra activities are included in that 
executable. 

*=> ACTION: Record app installation on client system 

o Have eStreamClientFileMgr record application nickname, DRM server 
name, & application serial number in database of apps installed on client. 



=> ACTION: Return status to AppInstallManager caller. 
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eStream 1.0 License Subscription Manager (LSM) 

Omnishift Technologies, Inc. 
Company Confidential 

Functionality 

This component is a COM Server executable. 

The LSM manages the users subscriptions to the different ASP accounts. It is part of the 
client component downloaded on a client machine. The LSM starts running when the 
client component starts running and is always active when the client component is 
running. Users on a given machine establish a connection with the ASP account servers 
from which they have subscribed applications. Users can add and delete the applications 
that are subscribed from the ASP accounts. The LSM makes the appropriate calls to the 
account servers to perform those actions. It gets serial numbers for the applications that 
are being subscribed and deletes them for the applications being un-subscribed (which are 
all part of the ASP ID Block). When the users start running any of the subscribed 
eStream applications, the eStream file system first queries the LSM before servicing any 
requests. The LSM in turn gets the appropriate access tokens from DRM servers along 
with the identities of application servers that can be used to run the applications. It uses 
the client identification (serial number) obtained when the connection to the ASP was 
made. At the same time, the LSM can decide to cache the access tokens and the identities 
of the application servers and decide to serve them directly from its cache. The eStream 
Cache Manager informs the LSM when applications start and end. The LSM keeps track 
of when access tokens are expiring and can request for additional access tokens when 
applications are running and the current one is about to expire. 



Data type definitions 

The global data managed by the LSM includes 

1 . The ASP ID Blocks which are obtained when the user on the machine establishes 
a connection with an ASP from which the user has subscribed applications. 



Field Name 


Type 


ASP ID 


GUE) 


ASP NAME 


BSTR 


ASP URL 


BSTR 


ASP IP 


DWORD 



2. The ASP Subscription Blocks are created when the user establishes an account 
with the ASP service. These blocks enables secure logon to the ASP Server. 



Field Name 


Type 


USER ED 


GUE) 


USER NAME 


BSTR 


USER HASH PASSWORD 


BSTR 



3. Application Subscription Blocks are created for every application that is 

subscribed. These blocks are created when the application subscription is started 
and are updated when the application is run. 



Field Name 


Type 


APPLICATION ID 


GUTD 


APPLICATION NAME 


BSTR 


RATE 


CURRENCY 


PERIOD 


INTEGER 



4. The access tokens and the identities of the applications servers that are obtained 
from the DRM servers when the user tries to run the applications. 



Field Name 


Type 


TOKENED 


GUTD 


APPLICATION ID 


GUED 


EXPIRATION 


DATE 


TOKEN SIZE 


DWORD 


TOKEN DATA 


BYTE* 



Interface definitions 



Subscription Management 

Subscription management is the main interface between the Client UI control panel and 
the License Subscription manager. Tables containing lists of Application Service 
Providers and Subscribed Applications are managed using the Subscription manager 
interface. 

ILicenseSubscritpionManager: rIDispa tch 

The LSM exposes the following set of APIs to the client UI. 

BOOL SubscribeApp(GUID & ASPId, GUID & AppID, Licenselnfo) 

This routine in turn will call the App Install Mgr to install the application on the client 
machine. This will return a Boolean stating success or failure. 

HRESULT UnsubscribeApp(ASPId, AppID) 

This routine will NOT implicitly uninstall the application. Applications must be explicitly 
uninstalled. This will return a Boolean stating success or failure. 

HRESULT GetNextApplD(GUID & AppID) 

This routine will return a pointer to a list of subscribed applications on the client 
machine. 

The LSM exposes the following set of APIs to the eStream file system. 
HRESULT CheckAccess(Path, &Root) 

The LSM establishes a co-relation between the Path and the AppID by querying the App 
Install Mgr. This routine in turn may contact the DRM server for appropriate access 
tokens. This will return a Boolean stating success or failure. At the same time root will 
get set to the head of the path that identifies the application so that the file system can use 
the same access token for everything under "root". 

BeginApp(ApplD) 

To indicate the start of an application. 
EndApp(ApplD) 

To indicate the end of the application. 



The LSM makes the following API calls. 

1 . lnstallApp(ASPId, AppID) to the App Install Mgr to install the subscribed 
applications. 

2. GetAppId(Path, &Root) to the App Install Mgr to get the Appld from the Path. 
"Root" is explained above. 

The LSM also sends messages to the account server for subscribing and unsubscribing 
applications and to the DRM server for getting access tokens. When a user goes to a new 
machine and installs the eStream client, the LSM obtains the subscription information 
from the account server when the user first establishes a connection with it. 



ISubscriptionManager 

ISubscription 

Methods Description 

SubscribeApp Subscribes an eStream Application. 

UnsubscribeApp Un-Subscribe an eStream Application 

CheckAccess Check to see if an eStream application is subscribed 

BeginApp Begin an eStream Application 

GetNextApp Get the next application the ASP Supplies 

GetNextSubscribedApp Get the next Subscribed Application 

GetLicenselnfo Get the license info for the application. 

HRESULT ISubscriptionManager:: SubscribeApp 

HRESULT SubscribeApp ( 
GUID AppID, 
GUID AspID, 

BSTR * license Info, ^ 
); 

Parameters 

AppID 

[in] Identifier for application to be subscribed. 

AspID 

[in] Identifier for the ASP service that the application is going to be subscribed to. 
licenselnfo 

[out] License info block for the subscribed application. 
Return Values 

Returns NOERROR if successful, or an OLE-defined error value otherwise. 



Remarks 

This function will return an error is a user attempts to subscribe an application that is 
already subscribed. 

HRESULT ISubscriptionManager:: UnSubscribeApp 

HRESULT SubscribeApp ( 
GUID AppID 
); 

Parameters 

AppID 

[in] Identifier for application to be un-subscribed. 
Return Values 

Returns NOERROR if successful, or an OLE-defined error value otherwise. 
Remarks 

This function will return an error is a user attempts to un-subscribe an application that 
not subscribed. 

BOOL ISubscriptionManager::CheckAccess 

HRESULT CheckAccess( 
GUID AppID 
); 

Parameters 

AppID 

[in] Identifier for application to be checked for access 
Return Values 

Returns TRUE if application can be access, FALSE otherwise. 



BOOL ISubscriptionManager:: BeginApp 



HRESULT BeginApp ( 
GUID AppID 
); 

Parameters 

AppID 

[in] Identifier for application to be started 
Return Values 

Returns S_OK if application can be started, ENOACCESS if the application is not 
subscribed. 

BOOL ISubscriptionManager:: GetNextApp 

HRESULT GetNextApp ( 
GUID AspID 
GUID AppID 
) ; 

Parameters 

AspID [in] ASP Account ID to check for application 
AppID 

[out] Identifier for application to be queried for ID this will be null when the list 
of applications runs out. 

Return Values 

Returns S_OK if application can be started, E_JNOACCESS if the application is not 
subscribed. 



Token Management - Overview 

Token management is the other major function that the eStream License Manager 
provides. These tokens are requested from the eStream Server every time an eStream 
application is started and release when an eStream application is terminated. Access 
tokens are issued for a finite period and renewed on a periodically depending on their 
expiration timestamp. 



Token Management - Cache Manager Interface 

Token management is the service that the License Manger provides to the cache manager. 
A table will be used store access tokens that the Cache manager uses. The 
ITokenManger interface provides access to tokens. 

ITokenManager::IDispatcb 

The LSM exposes the following set of APIs to the Cache Manage. 

HRESULT ITokenManager::GetToken 

HRESULf GetToken( 
GUID AppID, 
GUID AspID, 
DWORD expires, 
GUID & TokenID, 
DWORD Tokens ize, 
BYTE * Tokendata 
); 

Parameters 

AppID 

[in] Identifier for application to be subscribed 

AspID 

[in] Identifier for the ASP service that the application is going to be subscribed to 

Expires 

[out] Time interval for which the token is valid 
TokenID 

[out] Token ED 
TokenSize 

[out] Size of the token data 
Tokendata 

[out] token data 



Return Values 

Returns S_OK if successful, an expired token returns an error. 
Remarks 

Acquire a token for the License manager. 

HRESULT ITokenManager::ReleaseToken 

HRESULT ReleaseToken( 

GUID TokenID, 
); 

Parameters 

TokenID 

[in] ID for token to be released 

Return Values 

Returns S_OK if successful, an expired token returns an error. 
Remarks 

Release a token for the License manager. 



Format of ApplnstallBlock (part 1 of 2) 
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AIB Version 


AppID 
(GUID) 


AppID 
cont. 


Version No 


Client OS bitmap 


Client OS ServicePack 


Header 
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Sections 


Section 1 Type 
(Files) 


Section 1 Byte Offset 


Section 1 Byte Size 


Section 2 Type 
(Registry) 


Section 2 Byte Offset 


Section 2 Byte Size 


Section 3 Type 
(Prefetch) 


Section 3 Byte Offset 


Section 3 Byte Size 


Section 4 Type 
(Profile) 


Section 4 Byte Offset 


Section 4 Byte Size 


Section 5 Type 
(Comments) 


Section 5 Byte Offset 


Section 5 Byte Size 


Section 6 Type 
(Code) 
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Section 6 Byte Size 
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Format of ApplnstallBlock (part 2 of 2) 
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File Num 
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eStream ApplnstallBlock Low Level Design 



Sanjay Pujare and David Lin 
Version 0,2 



Functionality 

The ApplnstallBlock is a block of code and data associated with a particular application. 
This ApplnstallBlock contains the information needed to by the eStream client to initial- 
ize' the client machine before the eStream application is used for the first time. It also 
contains optional profiling data for increasing the runtime performance of that eStream 
application. 

The ApplnstallBlock is created offline by the eStream Builder program. First of all, the 
Builder monitors the installation process of a local version of the application installation 
program and records changes to the system. This includes any environment variables 
added or removed from the system, and any files added or modified in the system directo- 
ries. Files added to the application specific directory is not recorded in the Applnstall- 
Block to reduce the amount of time needed to send the ApplnstallBlock to the eStream 
client. Secondly, the Builder profiles the application to obtain the list of critical pages 
needed to run the application initially and an initial page reference sequence of the pages 
accessed during a sample run of the application. The ApplnstallBlock contains an op- 
tional application-specific initialization code. This code is needed when the default ini- 
tialization procedure is insufficient to setup the local machine environment for that par- 
ticular application. 

The ApplnstallBlock and the runtime data are packaged into the eStream Set by the 
Builder and then uploaded to the application server. After the eStream client subscribed 
to an application and before the application is run for the first time, the ApplnstallBlock 
is send by the server to the client. The eStream client invokes the default initialization 
procedure and the optional application-specific initialization code. Together, the default 
and the application-specific initialization procedure process the data in the Applnstall- 
Block to make the machine ready for eStreaming that particular application. 

Data type definitions 

The ApplnstallBlock is divided into the following sections: header section, variable sec- 
tion, file section, profile section, prefetch section, comment section, and code section. 
The header section contains general information about the ApplnstallBlock. The infor- 
mation includes the total byte size and an index table containing size and offset into other 
sections. In Windows version, the variable section consists of two registry tree structures 
to specify the registry entries added or removed from the OS environment. The file sec- 
tion is a tree structure consisting of the files copied to C drive during the application in- 
stallation. The profile section contains the initial set of block reference sequences during 
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Builder profiling of the application. The prefetch section consists of a subset of profiled 
blocks used by the Builder as a hint to the eStream client to prefetch initially. The com- 
ment section is used to inform the eStream client user of any relevant information about 
the application installation. Finally, the code section contains an optional program tai- 
lored for any application-specific installation not jcovered by the default eStream applica- 
tion installation procedure. In Windows version, the code section contains a Windows 
DLL. 

Here is a detailed description of each fields of the AppInstallBlock. 

Note: Little endian format is used for all the fields spanning more than 1 byte. Also, 
BlockNumber specifies blocks of 4K byte size. 

1. Header Section: 

The header section contains the basic information about that AppInstallBlock. This 
includes the versioning information, application identification, 

Core Header Structure: 

o AibVersion [4 bytes]: Magic number or appInstallBlock version number 
(which identifies the version of the appInstallBlock structure rather than the 
contents). 

o Appld [16 bytes]: this is an application identifier unique for each application. 
On Windows, this identifier is the GUBD generated from the 'guidgen* pro- 
gram. Appld for Word on Win98 will be different from Word on WinNT if it 
turns out that Word binaries are different between NT and 98. 

o VersionNo [4 bytes]: Version number. This allows us to inform the client that 
the appInstallBlock has changed for a particular appld. This is useful for 
changes to the AppInstallBlock due to minor patch upgrades in the applica- 
tion. 

o ClientOSBitMap [4 bytes]: Client OS supported bitmap or ID: for Win2K, 
Win98, WinNT and other future OSs we might support (it should be possible 
to say that this appInstallBlock is for more than one OS). 

o ClientOSServicePack [4 bytes]: We might want to store the service pack 
level of the OS for which this appInstallBlock has been created. Note that 
when this field is set we cannot use multiple OS bits in the above field Clien- 
tOSBitMap. 

o Flags [4 bytes]: Flags pertaining to AppInstallBlock 

■ Bit 0: Reboot - If set, the eStream client needs to reboot the 
machine after installing the AppInstallBlock on the client ma- 
chine. 

» Bit 1 : Unicode - If set, the string characters are 2 bytes wide 
instead of 1 byte, 
o HeaderSize [2 bytes]: Total size in bytes of the header section, 
o Reserved |32 bytes]: Reserved spaces for future. 
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o NumberOlSections [1 bytej: Number of sections in the index table. This de- 
termines the number of entries in the index table structure described below: 

Index Table Structure: (variable number of entries) 

o SectionType [1 bytes]: The type of data describe in section. 0=file section, 
Invariable section, 2==prefetch section, 3=profile section, 4=comment section, 
5=code section. 

o SectionOffset [4 bytesj: The offset from the beginning of the file indicates 

the beginning of section, 
o SectionSize [4 bytes]: The size in bytes of section. 

Variable Structure: 

o ApplicationNameLength [4 bytes]: Byte size of the application name 

o AppIicationName |X bytes]: Non-null terminating name of the application 

2. File Section: 

The file section contains a subset of the list of files needed by the application to run 
properly. This section does not enumerate files located in the standard application 
program directory. It consists of information about files copied into 'unusual' direc- 
tory during the installation of an application. If the file content is small, the file is 
copied to the client machine. Otherwise, the file is relocated to the standard program 
directory suitable for streaming. The file section data is list of trees stored in a con- 
tiguous sequence of address space according to the pre-order traversal of the trees. A 
node in the tree can correspond to one or more levels of directory. A parent-child 
node pair is combined into a single node if the parent node has only a single child. 
Parsing the tree from the root of the tree to a leaf node results in a fully legal Win- 
dows pathname including the drive letter. Each entry of the node in the tree consists 
of the following structure: 

Directory Structure: (variable number of entries) 

o Flags |4 byte]: Bit 0 is set if this entry is a directory 

o NumberOfChildren |2 bytes]: Number of nodes in this directory 

o DirectoryNameLength [4 bytes]: Length of the directory name 

o DirectoryName [X bytes]: Non-null terminating directory name 

Leaf Structure: (variable number of entries) 

o Flags 14 byte]: Bit 1 is set to 1 if this entry is a spoof or copied file name 
o FileVersion [4? bytes]: Version of the file GetFileVersionInfo() if the file is 
Win32 file image. Need variable file version size returned by GetFileVersion- 
lnfoSizeO. Otherwise use GetFileTime() to retrieve the file creation time, 
o FileNameLength |4 bytes]: Byte size of the file name 
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o DataLength [4 bytes): Byte size of the data. If spoof file, then data is the 

string of the spoof directory. Jf copied file, then data is the content of the file 
o FileName |X bytesj: Non-null terminating file name 
o Data |X bytes]: Either the spoof file name or the content of the copied file 

3. Add Variable and Remove Variable Sections: 

The add and remove variable sections contain the system variable changes needed to 
run the application. In Windows system, each section consists of several number of 
registry subtrees. Each tree is stored in a contiguous sequence of address space ac- 
cording to the pre-order traversal of the tree. A node in the tree can correspond to one 
or more levels of directory in the registry. A parent-child node pair is combined into 
a single node if the parent node has only a single child. Parsing the tree from the root 
of the tree to a leaf node results in a fully legal key name. The order of the trees is 
shown here. 



a. Registry Subsection: 

1. "KHCR": HKEYCLASSESROOT 

2. "HKCU": HKEYCURRENTUSER 

3. "HKLM": HKEYLOCALMACHINE 

4. "HKU": HKJEY USERS 

5. "HKCC": HKEY CURRENT CONFIG 



Tree Structure: (5 entries) 

o ExistFlag [1 byte]: Set to 1 if this tree exist, 0 otherwise, 
o Key or Value Structure entries |X bytes]: Serialization of the tree into 
variable number key or value structures described below. 



Key Structure: (variable number of entries) 

o KeyFlag [1 byte]: Set to 1 if this entry is a key or 0 if it's a value structure 
o NumberOfSubchild [4 bytes]: Number of subkeys and values in this key 
directory 

o KeyNameLengtb [4 bytes]: Byte size of the key name 
o KeyName [X bytes]: Non-null terminating key name 

Value Structure: (variable number of entries) 

o KeyFlag (1 byte]: Set to 1 if this entry is a key or 0 if it's a value structure 
o ValueType [4 byte]: Type of values from the Win32 API function 

RegQueryValueExO: REG_SZ, REGBINARY, REG_DWORD 

REG LINK, REG_NONE, etc. . . 
o ValueNameLength (4 bytes]: Byte size of the value name 
o ValueDataLengtb |4 bytes]: Byte size of the value data 
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o VaJueName [X bytesj: Non-null terminating value name 
o ValueData [X bytes]: Value of the Data 

In addition to registry changes, an installation in Windows system may involve 
changes to the ini files. The following structure is used to communicate the ini file 
changes needed to be done on the eStream client machine. The ini entries are ap- 
pended to the end of the variable section after the 5 registry trees are enumerated. 

b. INI Subsection: 

o NumFiles [4 bytes]: Number of INI files modified. 

File Structure: (variable number of entries) 

o FileNameLength [4 bytes]: Byte length of the file name 

o FileName [X bytes]: Name of the INI file 

o NumSection [4 bytes]: Number of sections with the changes 

Section Structure: (variable number of entries) 

o SectionNameLengtb |4 bytes]: Byte length of the section name 
o SectionName [X bytes]: Section name of an INI file 
o NumValues (4 bytes]: Number of values in this section 

Value Structure: (variable number of entries) 

o ValueLength [4 bytes]: Byte length of the value data 
o ValueData |X bytes]: Content of the value data 

4. Prefetch Section: 

The prefetch section contains a list of file blocks. The Builder profiler determines the 
set of file blocks critical for the initial run of the application. This data includes the 
code to start and terminate the application. It includes the file blocks containing for 
frequently used commands. For example, opening and saving of documents are fre- 
quently used commands and should be prefetched if possible. Another type of blocks 
to include in the prefetch section is the blocks associated with frequently accessed di- 
rectories and file metadata in this directory. The format of the data is described be- 
low: 

o FileNumber [4 bytes]: File Number of the file containing the block to pre- 
fetch 

o BIockNumber |4 bytes]: Block Number of the file block to prefetch 
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5. Profile Section: (not used in eStream 1.0) 

The profile section consists of a reference sequence of file blocks accessed by the ap- 
plication at runtime. Conceptually, the profile data is a two dimensional matrix: 
Each entry [row, column] of the matrix is the frequency a block row is followed by a 
block column. In any realistic applications of fair size, this matrix is very large and 
sparse. Proper data structure must be selected to store this sparse matrix efficiently in 
required storage space and minimize the overhead in accessing this data structure ac- 
cess. 

The section is constructed from two basic structures: row and column structures. 
Each row structure is followed by N column structures specified in the NumberCol- 
umns field. Note that this is an optional section. But with appropriate profile data, 
the eStream client prefetcher performance can be increased. 

Row Structure: (variable number of entries) 

o FileNumber [4 bytes] : File Number of the row block 
o BlockNumber [4 bytes]: Block Number of the row block 
o NumberCohimns [4 bytes]: number of blocks that follows this block. This 
field determines the number of column structures following this field. 

Column Structure: (variable number of entries) 

o FileNumber [4 bytes]: File Number of the column block 

o BiockNumber [4 bytes]: Block Number of the column block 

o Frequency [4 bytes]: frequency the row block is followed by column block 

6. Comment Section: 

The comment section is used by the Builder to describe this AppInstallBlock in more 
detail. 

o Comment [X bytes]: Null terminating comment string 

7. Code Section: 

The code section consists of the application-specific initialization code needed to run 
on the eStream client to setup the client machine for this particular application. This 
section may be empty if the default initialization procedure in the eStream client is 
able to setup the client machine without requiring any application-specific instruc- 
tions. On the Windows system, the code is a DLL file containing two exported func- 
tion calls: InstallQ, Um'nstallQ. The eStream client loads the DLL and invokes the 
appropriate function calls. 
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o Code [X bytesj: Binary file containing the application-specific initialization 
code. On Windows, this is just a DLL file. 

8. LicenseAgreement Section: 

The Builder creates the license agreement section. The eStream client displays the li- 
cense agreement text to the end-user before the application is started for the first time. 
The end-user must agree to all licensing agreement set by the software vendor in or- 
der to use the application. 

o LicenseAgreement [X bytes]: Null terminating license agreement string 



Open Issues 

o What is the size of the ApplnstallBlock for a typical application like Office? 
o How large should the prefetch sections be for optimal run of an application? 

At minimum, it should contain at least start/termination code, 
o How should the ApplnstallBlock handle application license agreement text 

string? Add a new section or use comment section. Does the dialog need to 

have exactly the same interface as the license agreement dialog on the local 

installation? 

o Currently, file section stores complete pathname including the drive letter. 
The installation may place files according to some variables like %System- 
Root% or %UserProfile%. How does the Builder detect this so it can propa- 
gate this information to the client? 
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The eStream Builder 



The eStream Builder is a software program. It is used to convert locally installable 
applications into a data set suitable for streaming over the network. The streaming- 
enabled data set is called the eStream Set. This document describes the procedure used to 
convert locally installable applications into the eStream Set. 

The application conversion procedure into the eStream Set consists of the several steps. 
In the first phase, the Builder program monitors the installation process of a local 
installation of the desired application for conversion. The Builder monitors any changes 
to the system and records those changes in an intermediate data structure. After the 
application is installed locally, the Builder enters the second phase of the conversion. In 
the second step, the Builder program invokes the installed application executable and 
obtains sequences of frequently accessed file blocks of this application. Both the Builder 
program and the eStream client software use the sequence data to optimize the 
performance of the streaming process. Once the sequencing information is obtained, the 
Builder enters the final phase of the conversion. In this step, the Builder gathers all data 
obtained from the first two phase and processes the data into the eStream Set. 

In the next sections, detailed descriptions of the three phases of the Builder conversion 
process are described. The three phases consists of installation monitoring, application 
profiling, and finally eStream packaging. In most cases, the conversion process is 
general and applicable to all type of system. In places where the conversion is OS 
dependent, the discussion is focused on Microsoft Windows environment. Issues on 
conversion procedure for other OS environment are described in later sections. 

Installation Monitoring 

In the first phase of the conversion process, the Builder Installation Monitor (EM) 
component invokes the application installation program that installs the application 
locally. The IM observes all changes to the local computer during the installation. The 
changes may involve one or more of the following: changes to system or environment 
variables; and modifications, addition, or deletion of one or more files. The IM records 
all changes to the variables and files in a data structure to be sent to the Builder's eStream 
Packaging component. In the following paragraphs, detailed description of the 
Installation Monitor is described for Microsoft Windows environment. 

In Microsoft Windows system, the Installation Monitor (IM) component consists of a 
kernel-mode driver subcomponent and a user-mode subcomponent. The kernel-mode 
driver is hooked into the Windows registry and file system function interface calls. The 
hook into the registry function calls allows the IM to monitor system variable changes. 
The hook into the file system function calls enables the IM to observe file changes. 

The IM kernel-mode (IM-KM) driver subcomponent is controlled by the user-mode 
subcomponent (IM-UM). The IM-UM sends messages to the IM-KM to start and stop 
the monitoring process via standard I/O control messages called 10CTL. The IM-KM 
memorizes any addition or deletion of registry variables. It also records changes to 



application-specific, shared among a group of applications, or system-wide files. Every 
files and directories are assigned a unique file number for simplifying identification of a 
specific file. Once the installation of an application completed, the IM-UM retrieves 
these changes from the IM-KM and forward the data structure to the eStream Packager. 
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Application Profiling 

In the second phase of the conversion process, the Builder's Application Profiler (AP) 
component invokes the application executable program that is installed during the first 
phase of the conversion process. The executable program files are accessed in a 
particular sequence. And the purpose of the AP is to captures this sequence data. This 
data is useful in several ways. 

First of all, frequently used file blocks can be streamed to the eStream client before other 
less used file blocks. A frequently used file blocks is cached locally on the eStream 
client cache before the user starts using the streamed application for the first time. This 
has the effect of making the streamed application as responsive to the user as the locally 
installed application by hiding any long network latency and bandwidth problems. 

Secondly, the frequently accessed files can be reordered in the directory to allow faster 
lookup. This optimization is useful for directories with large number of files. When the 
eStream client looks up a frequently used file in a directory, it finds this file early in the 
directory search. In an application run with many directory queries, the potential 
performance gain is significant. 

The Application Profiler (AP) is not as tied to the system as the Installation Monitor (IM) 
but there is still some OS dependent issue. In the Windows system, the AP still has two 
subcomponents: kernel-mode (AP-KM) subcomponent and the user-mode (AP-UM) 
subcomponent. The AP-UM invokes the converting application executable. Then AP- 
UM starts the AP-KM to track the sequences of file block accesses by the application. 
Finally when the application exits after the desired amount of sequence data is gathered, 
the AP-UM retrieves the data from AP-KM and forwards the data to the eStream 
Packager. 
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EStream Packaging 

In the final phase of the conversion process, the Builder's eStream Packager (EP) 
component processes the data structure from IM and AP to create a data set suitable for 
streaming over the network. This converted data set is called the eStream Set and is 
suitable for uploading to the eStream Servers. 



The eStream Set consists of the three sets of data from the eStream Server's perspective. 
The three types of data are Concatenation Application File (CAF), Size Offset File Table 
(SOFT), and Root Versioning Table (RVT). 

The Concatenation Application File (CAF) consists of all the files and directories needed 
to stream to the client. The CAF can be further divided into two subsets: initialization 
data set and the runtime data set. The initialization data set is the first set of data to be 
streamed from the server to the client. This data set contains the information captured by 
IM and AP needed by the client to prepare the client machine for eStreaming this 
particular application. This initialization data set is also called the ApplnstallBlock. 
Detailed format description of the ApplnstallBlock is described in another document. 
The second part of the CAF consists of the runtime data set. This is the rest of the data 
that is streamed to the client once the client machine is initialized for this particular 
application. The EP appends every files recorded by IM into the CAF and generates all 
directories. Each directory contains list of file name, file number, and the metadata 
associated with the files in that particular directory. 

The EP is also responsible for generating the SOFT file. This is a table used to index into 
the CAF for determining the start and the end of a file. The server uses this information 
to quickly access the proper file within the directory. 

Finally, the EP creates the RVT file. The Root Versioning Table contains a list of root 
file number and version number. This information is used to track minor application 
patches and upgrades. The EP generates new directories when any single file is changed 
from the patch upgrade. The RVT is uploaded to the server and requested by the eStream 
client at appropriate time for the most updated version of the application by a simple 
comparison of the client's eStream application root file number with the RVT table 
located on the server. 
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Data Flow Description 

The following list describes the data that is passed from one component to another. The 
numbers corresponds to the numbering in the Data Flow diagram. 

1 . The full pathname of the installer program is query from the user of the Builder 
program and is sent to the Install Monitor. 



2. The Install Monitor (IM) user-mode sends a read request to the hard-drive 
controller to spawn a new process for installing the application on the local 
machine. 

3. The OS loads the application installer program into memory and run the installer 
program. 

4. The installer program reads more files from the CD media. 

5. The CD media data files are read into memory by the installer program. 

6. The application installer program writes the files into proper locations on the local 
hard-drive. 

7. IM kernel-mode captures all file read/write requests and all registry read/write 
requests by the installer program. 

8. IM kernel-mode program sends the list of all file changes and all registry changes 
to the IM user-mode program. 

9. IM user-mode identify special files which needs to be copied or spoofed into 
eStream client machine before the regular files can be streamed. It also assigns 
unique file numbers to every file. This data is returned to the Builder UI. 

1 0. Builder UI invokes Application Profiling (AP) user-mode program by querying 
the user for the list of application executable names to be profiled. 

1 1. Application Profiler user-mode invokes each application executable in succession 
by spawning each program in a new process. 

12. The OS loads the application executable into memory and run. the executable. 

1 3. The executable file image is loaded into memory and starts executing. The 
application files will continuously be loaded into memory as needed. 

14. Every file accesses to load the application file blocks into memory is monitored 
by the Application Profiler (AP) kernel-mode. 

1 5. Application Profiler kernel-mode returns the file access sequence and frequency 
information to the user-mode program. 

1 6. Application Profiler returns the processed profile information. This has two 
sections. The first section is used to identify frequency of files accessed. The 
second section is used to list the file blocks for prefetch to the client. 

17. The eStream Packager receives files and registry changes from the Builder UI. It 
also receives the file access frequency and a list of file blocks from the Profiler. 

1 8. The eStream Packager reads all file data from the hard-drive that are copied there 
by the application installer. 

1 9. The eStream Packager also reads the previous version of eStream Set for support 
of minor patch upgrades. 

20. Finally, the new eStream Set data is stored back to non-volatile storage. 

Mapping of Data Flow to eStream Set 

Step 7: Data gathered from this step consist of the registry and file changes. This data is 
mapped to the ApplnstallBlock's File Section, Add Registry Section, and Remove 
Registry Section. 

Step 8 & 19: File data are copied to the local hard-drive then concatenated into part of the 
CAF contents. 

Step 10: Data returned to the Builder UI contains unique file numbers. This data is 
mapped to the file numbers used throughout the eStream Set data structure. 



Step 15: Part of the data gathered by the Profiler is used to generate a more efficient 
eStream FS Directory content. Another part of the data is used in the 
ApplnstallBlock as a prefetch hint to the eStream client. 

Step 20: If the installation program was an upgrade, eStream Packager needs previous 
version of the eStream Set data. Appropriate data from the previous version is 
combined with the new data to form the new eStream Set. 
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Format of eStream Set 

The format of the eStream Set consists of 3 sections: Root Version Table (RVT), Size 
Offset File Table (SOFT), and Concatenation Application File (CAF). The RVT section 
lists all versions of the root file numbers available in an eStream Set. The SOFT section 
consists of the pointers into the CAF section for every file in the CAF. The CAF section 
contains the concatenation of all the files. The CAF section is made up of regular 
application files, eStream FS directory files, AppInstallBlock, and icon files. Please see 
the document on eStream Set Format for detailed format of the eStream Set. 

OS dependent format 

The format of the eStream Set is designed to be as portable as possible across all OS 
platforms. At the highest level, the format of CAF, SOFT, and RVT that make up the 
format of eStream Set are completely portable across any OS platforms. The only critical 
piece of data structure that is OS dependent is located in the initialization data set called 
AppInstallBlock in the CAF. This data is dependent on the type of OS due to the 
differences in low-level system differences among different OS. For example, the 
Microsoft Windows contain system environment variables called the Registry. The 
Registry has a particular tree format not found in other operating systems like UNIX or 
MacOS. 

Another OS dependent format is the format of the file names. Applications running on 
the Windows environment inherit the old MSDOS 8.3 file name format. To support this 
properly, the format of the Directory file in CAF requires an additional 8.3 field. This 
field is not needed in other operating systems like UNIX or MacOS. 
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Device driver versus file system paradigm 

The eStream Prototype is implemented using the 'device driver' paradigm. One of the 
advantages of the device driver approach is that the caching of the sector blocks is 



simpler. The client cache manager only needs to track sector number in its cache. In 
comparison with the 'file system' paradigm, more complex data structure is required to 
track a subset of a file that is cached on a client machine. This makes 'device driver' 
paradigm easier to implement. 

On the other hand, there are many drawbacks to the 'device driver' paradigm. On the 
Windows system, the device driver approach has problem supporting large number of 
applications. This is due to the limitation on the number of assignable drive letters 
available in a Windows system (26 letters); and the fact that each application needs to be 
located in its own device. Note that having multiple applications in a device is possible, 
but then the server needs to maintain exponential number of devices that support all 
possible combinations of applications. This is too costly to maintain on the server. 

Another problem with the device driver approach is that the device driver operates at the 
disk sector level. This is a much lower level than operating at the file level in the file 
system approach. The device driver does not know anything about files. Thus, the 
device driver cannot easily interact with the file level issues. For example, spoofing files 
and interacting with OS buffer cache is nearly impossible with device driver approach. 
But both spoofing files and interacting with OS buffer cache is need to get higher 
performance. 
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Implementation in the Prototype 

The prototype has been implemented and tested successfully on the Windows and Linux 
distributed system. The prototype is implemented using the 'device driver' paradigm as 
described above. The exact procedure for streaming application data is described next. 

First of all, the prototype server is started on either the Windows or Linux system. The 
server creates a large local file mimicking large local disk images. Once the disk images 
are prepared, it listens to TCP/IP ports for any disk sector read or write requests. 

Secondly, the conversion process is done on a Windows system via semi-manual 
procedure. The server disk image is 'mounted' on the local Z drive by making the proper 
TCP/IP connection to the server. Then the application installation program is invoked 
and the application is installed into the Z drive. This writes the application files into the 
Z drive device driver, through the TCP/DP connection, and finally on to the server disk 
image. At the same time, a file and registry monitoring program records all registry and 
file changes. This data is stored as an initialization file to be invoked on the client to 
prepare the client machine for streaming. 

Finally, after the application files is stored on the server disk image, the client prototype 
is started. The client connects to the server and 'mount' the server disk image as a local 
Z drive. Then the initialization file is invoked which setup the local registry variables 
and copy system files into proper directories. Once the local machine is prepared for 
streaming that particular application, the user can start using the application. When the 
application is first started, the pages are not located in the local buffer cache. The OS 
makes sector request to the eStream device driver that forwards the sector request to the 
eStream Cache Manager. If the sector is located in the eStream cache, then the data is 
returned immediately. If the data is not located in the eStream cache, then the request 
forwarded to the network component that sends the message to the server. The server 
finds the proper sector data and returns the data to the client. The client eStream Cache 
Manager caches the new sector data and forwards the sector data to the eStream device 
driver. The device driver returns the sector data to the OS. 



eStream Builder Data Flow Diagram 



Builder Ul 



App 
CD 



10 



Install 
Monitor 
User-mode 




Install 



16 



Application 

Profiler 
User-mode 



Local 
HD 



11 



12 



13 




-M Execution 



18 



17 



eStream 
Packager 



20 



eStream 
Set 



19 



USER 



14 



15 



KERNEL 



Install 
Monitor 
Kernel-mode 



Application 

Profiler 
Kernel-mode 



v0.1 



Restore Data 
From Before 
Reboot 



Builder Install Monitor Control Flow Diagram 




Query 
User for 
Install Program 
Pathname 



Start IM 
Kernel-Mode 



Start 
Install Monitor 
.Kernel- mode 
Driver 



Start 
Signal 



Start 
Monitoring 
Registry & File 
Changes 




Builder Profiler Control Flow Diagram 



Start AP 
User-Mode 



Query 
User for List of 
Executable 

Program 
Pathnames 



Start AP 
Kernel-Mode 



Start 
Profiler 
Kernel-mode 
Driver 



Start 
Signal 



Wait 
for Start 
Signal From 
User-mode 
Program 



Invoke Each 
Executable 
Program 



Wait for 
a Process 
to Terminate 


No 


< < 











Yes 








Return 


Data & Status 


to Builder Ul 


> 





Yes 



Tell Kernel-mode 
Driver to Stop 
Gathering Data 



End 




Record 
File Access 
Counter & 
File Block 



Stop 
Signal 



Wait for 
a Change in 
the System 
Or to Stop 
Monitoring 



/V 




Get Data 
From 
Kernel-mode 
Driver 



Data 



Yes 



Send Data 
To 

User-mode 
Program 



Builder eStream Packager Control Flow Diagram 



Start EP 



Get Registry 

and 
File Changes 
From Ul 



Get File Access 

Frequency 
and Prefetch File 
Blocks 



Generate 
ApplnstallBlock 
from Old Version 
and New Data 



Get Previous 
Version of 
eStream Set 



Yes 




No 



Generate 
Updated 
eStream FS 
Directory Files 



Append Updated 
File Contents 
to End 
ofCAF 



Update 
SOFT 



Generate 
New 
eStream Set 

From CAF, 
SOFT, RVT 



Update 
RVT 



Generate 
ApplnstallBlock 
from New Data 




t 


Generate 
eStream FS 
Directory Files 




r 


Append All 
File Contents 
to CAF 




f 


Create 
SOFT 







End 



Create 
RVT 



estream Application Builder High-Level Design Diagram 



User Interface 



Builder Ul 



App 
CD 



A1 



A2 



A3 



f t 



C1 



Merge 
Module 



B1 



A7 



Install 
Monitor 
Module 



Manual 
Intervention 
(optional) 




A6 



B5 



Profile 
Manager 
Module 



A5 



B2 



B6 



C2 



Profile 
Data 



C3 



C4 



Packaging 
Manager 
Module 




B3 



A9 B8 



C5 



eStream 
Set 



B7 



B4 



USER 



KERNEL 



A4 



FSRFD 



Registry 
Monitor 



File Filter 
Monitor 



A4 



File 
Access 
Monitor 



A4 



B3 




v0.1 



eStream Builder File Access Monitor Low Level Design 



Sanjay Pujare and David Lin 
Version 0.1 



Functionality 

The eStream Application Builder File Access Monitor (FAM) is a kernel-mode device 
driver that behaves as a file filter driver to has the following responsibilities: 

□ Monitor any running application's request to access a file or directory 

□ Track application file and directory accesses 

□ Track file metadata queries 

□ Start and stop profiling via IOCTL requests from the user-mode program 

□ Return the file access data to the user-mode program via I/O Request Packet 
(IRP) 

□ Return any error conditions to the user-mode program via IRP 

The File Access Monitor is based on the Tilemon' program. The source code for the 
program is available free for download over the Web at 
http://www.sysinternals.com/filemon.htm. 

Data type definitions 

The File Access Monitor (FAM) monitors a sequence of file block accesses by a particu- 
lar process or one of its child processes. The FAM also tracks any queries on the file 
metadata. The combination of the file content and metadata is returned to the Profile 
Manager for further processing. The following is the data structure externally visible to 
the other subcomponents outside FAM. 

Struct SequenceData 
{ 

UINT NumEntries; 
Struct Entry 

{ 

PUNICODE_STRING FilePathName ; 
BOOL IsAccessingMetadata; 
ULONG Offset; 
ULONG Size; 
} Entries [NumEntries] ; 

}; 

The FileName contains the null terminating string of the file accessed. IsAccessMetadata 
flag indicates if the access is on the file metadata or the file content. If the operation is on 
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the file content, then the fields 'Offset' and 'Size' indicate the location of the read or 
write operations. Otherwise, the fields 'Offset' and 'Size' are not used. 

Interface definitions 

Function 1 : Hooks into user defined IOCTL calls 

// The following is a Fast I/O Device Control 
// call interface. Each of the user IOCTL 
// call to the driver is described here. 
// The OICTL input and output parameters are 
// stored on the IRP on the InputBuffer and 
// OutputBuffer respectively. 

// This function must be called only by NT I/O 
// Manager. 

BOOLEAN FileSysFastloDeviceControl ( 
IN PFILE_OBJECT FileObject, 
IN BOOLEAN Wait, 
IN PVOID InputBuffer, 
IN ULONG InputBuf ferLength, 
OUT PVOID OutputBuffer, 
IN ULONG OutputBuf ferLength, 
IN ULONG IoControlCode, 
OUT PIO_STATUS_BLOCK IoStatus, 
IN PDEVICE_OBJECT DeviceObj ect ) 

Input : 

I oCon t r o 1 Code = = I OCTL_FAM_VERS I ON 

OutputBuffer: version number of the driver 

I oCon t r o 1 Code = = I OCTL_FAM_S TART 

InputBuffer: process ID to monitor 

IoControlCode==IOCTL_FAM_STOP 
OutputBuffer: stop profiling 

I oCon t rol Code = = I OCTL_FAM_GETDATA 
OutputBuffer: get sequence data 

I oCont rol Code = = IOCTL_FAM_GETSTATUS 

OutputBuffer: get status from driver 



Output : 
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Comments : 

The IOCTL calls from the user program to 
the device driver is either through the 
dispatcher or through the Fast I/O 
interface. 

Errors : 

Function 2: DriverEntry 

// Called by the NT system to initialize 

// driver. The following entries are hooks 

// into the OS and are not called by any of our 

// component directly. 

NTSTATUS DriverEntry ( 

IN PDRIVERJDBJECT DriverObj ect , 
IN PUNICODEJSTRING RegistryPath) 

Comments : 

Initialize the driver 

Function 3: Hooks into Fast I/O functions 

// NT Fast I/O calls. These are some of the 
// hooks into the OS 
NTSTATUS FileSysFastIoRead( 

IN PDRIVER_OBJECT DriverObj ect , 

IN PLARGE_INTEGER FileOffset, 

IN ULONG Length, 

IN BOOLEAN Wait, 

IN ULONG LockKey, 

OUT PVOID Buffer, 

OUT PIO_STATUSJBLOCK IoStatus, 

IN PDEVICE_OBJECT DeviceObj ect ) 

Comments : 

Hooks into Fast I/O Read 

Function 4: Hooks into dispatcher functions 

// Besides hooks into Fast I/O calls, we 
// must also hook into each of the major 
// functions like IRP_MJ_CREATE, IRP_MJ_READ, 
// etc... 

FileSysHookRoutine ( 

PDEVICE_OBJECT HookDevice, 
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IN IRP Irp) 



Component design 

I/O Hook location 

The trickiest part of the File Access Monitor (FAM) component design is determining the 
locations in the Operating System to hook the routines. FAM must provide driver entry 
function for initializing the driver. It must also provide hooks into the OS to monitor 
read and write file operations. In addition, it needs to monitor access to file metadata. 
And finally, it has to provide the user-mode program a way to communicate to the FAM 
through the IOCTL calls. 

The FAM must behave like any other Windows kernel-mode drivers by exporting the 
standard DriverEntry function. The DriverEntry function has the following purposes: 

o Check for OS build version. 

o Setup the device name 

o Call OS routine to create the device 

o Make symbolic link to allow device access from Win32 programs 

o Create dispatch points for all routines that must be handled 

o Setup Fast I/O hooks 

o Initialize all data structures 



In addition to the DriverEntry, the FAM must handle five user defined IOCTL calls. 

o IOCTL_FAM_VERSION 

o IOCTLFAMSTART 

o IOCTL_FAM_STOP 

o IOCTLFAMGETDATA 

o IOCTLFAMGETSTATUS 



In IOCTL FAM START, the handler receives the process ID from the user-mode pro- 
gram. It uses this process ID to filter out relevant file and metadata accesses. In 
IOCTLFAMSTOP, the handler stops monitoring and recording any file accesses. In 
IOCTL FAM GETDATA, the handler packages the file access sequence in the I/O Re- 
quest Packet (IRP) to be returned to the user-mode program. Finally, in 
IOCTL FAM GETSTATUS, the handler returns its current status. This status includes- 
FAMSTATUSOK, FAMSTATUSERROR, and FAMSTATUSPROFILING. 

In addition to the user defined IOCTL hooks, the FAM must add hook into both the dis- 
patch points and the Fast I/O calls to monitor all read and write requests. In addition, the 
FAM monitors any metadata accesses. The following is a list of Fast I/O calls it must 
hook: 



o FastloRead 
o FastloWrite 
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o FastloMdlReadComplete 

o FastloMdlWriteComplete 

o FastloReadCompressed 

o FastloWriteCompressed 

o FastloQueryBasicInformation 

o FastloQueryStandardlnformation 

In the routine to handle FastloRead and FastloWrite, the driver must determine the proc- 
ess ID making this request. If the process is in the list of monitoring processes, the file 
name, file offset, and size is recorded and added to the profile sequence list. In the rou- 
tine to handle FastloQueryBasicInformation and FastloQueryStandardlnformation, the 
driver records the file name associated with this metadata query. 

In addition to hooks to the Fast I/O calls, the I/O may call the File System services 
through standard Windows NT dispatch points. The following is a list of dispatch points 
to be handled by FAM: 

o IRPMJCREATE 

o ERP MJ READ 

o IRPJV1J_WRITE 

o IRP_MJ_DIRECTORY_CONTROL + IRP MN QUERY DIRECTORY 

o IRP MJ QUERY MFORMATION 

o IRPJVIJ_SET_INFORMATION 

o IRP_MJ__QUERYJ3A 

o IRP_MJJSET_EA 

The routine to handle IRP_MJJRJBAD, IRPJV1J WRITE, and 
IRP_MN_QUERY_DIRECTORY is handled by the same function as the routine for 
handling FastloRead and FastloWrite. The routine to handle 
IRPJVIJQUERYJNFORMATION, IRP_MJ_SETJNFORMATION, 
IRPMJ_QUERY_EA, and IRPMJ^SETJEA are handled by the same function as the 
routine for handling FastloQuerylnformation. 

Communication with user-mode component (Profile Manager) 

Besides using the IOCTL to send profile data to the Profile Manager, the FAM must also 
signal the Profile Manager when new data is available for retrieval. The Profile Manager 
wakes up from the signal by the FAM and retrieves the information on the blocks of files 
accessed. FAM also signals the profile manager when the profiled application termi- 
nates. FAM uses KeSetEventQ to send a 'data available 7 event signal to the profile man- 
ager. Profile manager calls KeWaitForSingleEventQ or KeWaitForMultipleEventQ to 
wait for a signal from the kernal-mode driver. KeClearEventQ is called by the FAM 
when the signal to profile manager should be deactivated. 

Process Filtering 

The FAM must filter the profile information so only relevant data relating to the applica- 
tion under profiled is obtained. This is accomplished by filtering the data according to 
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^reaterrocessQ API. Thus, the new processes all inherit Profile Manager as if, n*™t 

RoT m ^ TT fi,lerUlg " accom P^hed using P^O^S^^ST 
RoutmeO to add a hook to the OS. FAM is notified whenever tu» errocesmotl Jy- 
aled The process JD is recorded in a lis, ^T^TZZ^M^'ZIZ 
This hst is used to filter the profile data gathered by FAM. iVJana ger process ID. 



Locks 



and thai KQL must be lower than DISPATCH ! LEVEI t 7 Ca " mg 

ERESOURCE gResource; // global variable 
KeEnterCriticalRegionO 

ExAcquireResourceExclusiveLite( & gResource, TRUE) ; 
<critical section of code> 

ExReleaseResourceLitef&gResource) • 
KeLeaveCriticalRegionO ; 

Testing design 

o Unit testing plans 

The plan for unit testing of the FAM consists of using the Profile Manaeer tPMl 

roc^ Z£z D F n A v n * A ?>? 1,16 ,esl drivers PM 

iul i i. calls. The FAD creates desired data pattern from the OS 'c T/n iu 

IIS J^f^ t6StS the FAM ' S *4 tolnr file ™ fry- 
ing files and directories m a particular order. Together the PM and FAD tit 
coverage of the FAM is complete. The following^ ^ ^ 

1 . Test each user-defined IOCTL interface via PM by sending border cases 

file vZt: Tl FAM CaPtUr6S 6Very fi,c a " d ^ •Sccs^.^Sarf 
file I/O requests from a user-mode program called FAD. 
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o Stress testing plans 



o Coverage testing plans 



o Cross-component testing plans 

Cross-component testing for the Builder program is described in the Package 
Manager low-level design document. 



Upgrading/Supportability/Deployment design 

Other Builder components log error messages to a predefined file. The kernel-mode pro- 
grams do not have the capability to read/write to a file. Since FAM is a kernel-mode 
program, an alternative method of reporting error messages has to be developed. Current, 
the FAM has a user-defined IOCTL interface (IOCTL FAM GETSTATUS) to retrieve ' 
the error messages. FAM keeps a stack of error messages encountered and reports the 
stack of error messages at the request by an appropriate user-mode program. 

Open Issues 



o Exactly which Fast I/O calls need to be hooked to get all the read and write opera- 
tions for file accesses? 

o Along the same line, which dispatch points need to handled to get all the read and 
write operations for file accesses? 

o Have we hooked into all possible places where the metadata accesses can occur? 

o Does the FAM need to hook into FileLock and FileUnlock operations? 
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This document contains the high level design of the eStream Application Builder. The 
Builder is used to "prepare" an application before it can be eStreamed. This document 
describes the high level design of the application installation monitoring, file relocation 
and mapping, gathering of the initial profiling information of an application, the 
packaging of the eStream Set, and the merging of the newly uploaded user profile data. 

Note: all references to "user" should be understood to mean the user of the Builder (i.e. 
the person who is responsible for creating eStream sets) and not the end-user of eStream 
technology. 

This document described these steps involved in the preparation of the application: 
Installation Monitoring, Application Profiling, and eStream Packaging. 

Modules 

Installation Monitor: 

• When the application is installed, we need to monitor the installation to see 
various "things" taking place on the computer. These could be: 

■ Various updates to the System Registry 

• Files added to the Install directories (i.e. directories where application bits are 
copied as specified by the installing user). Lets call this group Fj. 

■ Files added/updated to the Shared directories (e.g. "Program Files\Common 
Files"). Lets call this group F c . 

■ Files added/updated to the System directories (e.g. "WinNT\System32"). Lets 
call this group Fs. 

■ Files added/updated to the User specific directories (e.g. "Documents and 
Settings\spujare\Application Data"). Lets call this group Fu. 

Note that once this information is gathered by the "Installation Monitor", a single 
"Installation Set" is prepared where all the files are stored in a single directory 
hierarchy. Note that files in the F c , F s and Fu groups (i.e. F C su group) are also 
stored here. For these files a "mapped location" is created under the single 
directory hierarchy. The Installation Set typically creates a map of all files (called 
ISM for Installation Set Map) described above with each entry containing the 
following info: 

1. fileld for the file 

2. location and name of the file. Note that the location will be the actual location 
for F] files, but mapped location for the F C su files. 



• After we gather the above information, we need to prepare a "File Relocation 
Map" (FRM) that is used by the client file spoofer to spoof references to any file 
in the common file group (i.e. F C su)* For example: when the eStreamed app 
makes a reference to a file C : \Program Files\Word\Foobar, the file 
spoofer actually redirects that reference to Z : \ Program 
Files\Word\Foobar. It does that because of the File Relocation Map. Each 
entry in the FRM typically has the following info: 

1 . fileld (which references an entry in the ISM). 

2. Actual location where the application expects it (i.e. C : \ Program 
Files\Word\Foobar) . 

Profile Module: 

During the application building process, the Builder program queries the user for the 
name of the application executable. Then Builder program starts and terminates the 
application executable immediately to gather initial sequence of the application page 
access pattern. After the initial seed of profile data is acquired, the Profile Sequence 
Matrix is combined with other applnstallBlock data gathered from the Install Monitor. 

Profile Sequence Matrix is a 2D matrix of a profile data. Each entry of the matrix 
[column C, row R] is an integer value indicating the number of times a page R is 
requested following the request of page C. This successor request pattern is the page 
requests missed in the eStream cache manager. 

Package Module: 

In the final phase of the Builder program, the applnstallBlock is encapsulated into a 
special installation executable and the application files is archived into a single 
compressed package. The install executable containing the applnstallBlock and the 
archive of application files can then be placed in a suitable eStream Set server for ASP to 
download to their machines. 

Merge Module: (not supported in version 1.0) 

During normal eStream application usage, the eStream client gathers profile information 
for that particular run of the application. Then at the termination of an eStream 
application, it uploads the new Profile Sequence Matrix to the Profile Server. The clients 
should not upload the Profile Sequence Matrix from previous runs because the Profile 
Server has no mechanism for distinguishing between previously uploaded data and the 
newly acquired data. 

At appropriate time, the Builder is invoked to merge the newly uploaded per-user Profile 
Sequence Matrix into a collective Matrix. The merging algorithm may be designed with 
some heuristics to prevent the data biasing toward power users. This collective Matrix 
can be reinserted into the appropriate applnstallBlock then downloaded by any requesting 
eStream clients. 

Kernel Device Drivers: 



In addition, kernel device drivers are used to actually hook into the operating system to 
monitor the registry and file changes during installation of the application. This is 
accomplished by the FSRFD module. 

The kernel device driver is also used for gathering monitoring file block references from 
the operating system to the file system. This is accomplished by the File Access Monitor. 
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User Interlace 




Interfaces 



The interfaces are divided into three use cases: application installation monitoring, 
application profiling, and merging of the uploaded profile data. 

Use Case #1 : Install Monitor 

Al . Builder UI to Install Monitor - send the name of the application installation 
executable 

A2. App CD to Install Monitor - the CD containing the application is fed into the 

installation monitor module 
A3. Install Monitor to Installation App - invoke the installation program 
A4. Installation App to FSRFD - monitor all changes to the registry and files when 

installation program write to local file system 
A5. FSRFD to Install Monitor - send all registry and file changes 
A6. Install Monitor to itself - repeat all applications in the suite and merge all data 
A7. Install Monitor to Manual Intervention - send a list of registry and file captured 

by the install monitor to UI and allow user to add or delete any entries 
A8. Manual Intervention to Package Manager - send the final registry and file 

relocation data to the packager 
A9. Package Manager to database - data set is packaged into appInstallBlock and the 

rest of the application files suitable for eStreaming 

Use Case #2: Profiling 

BL Builder UI to Profile Manager - send the name of the application executable 
B2. Profile Manager to Run App - invoke the application 

B3. Run App to eStream File Access Monitor - record sequences of page requests 
B4. File Access Monitor to Profile Manager - save the profile information 
B5. Profile Manager to Profile Manager - repeat for each application in the suite 
B6. Profile Manager to Package Manager - send all profile data for merging into a 
single data 

B7. database to Package Manager - get eStream Set from the database 
B8. Package Manager to database - save the updated eStream Set 

Use Case #3: Merging Profile data (not supported in version LO) 

CI . Builder UI to Merger - send the application name with profile data to merge 
C2. database to Merger - get uploaded Profile Sequence Matrix from the Profile 
Server 

C3. database to Merger - get the old appInstallBlock from database 

C4. Merger to Package Manager - reinsert the Profile data into appInstallBlock 

C5. Package Manager to database - save the updated appInstallBlock 

Requirements 

Please see eStream 1 .0-REQ.doc for the most up-to-date list of the Builder requirements. 
This requirement list may not contain the most recent changes. Each requirement is 
identified by a lag such as R-XXXX for easy references elsewhere in the document. 



R-Background: The installation monitor runs in the background, when an 
eStream application is installed as part of its preparation or building. 
R-RegistryCapture: The installation monitor captures all the updates to the 
System Registry that take place during the install. These updates are captured as a 
.REG file. Note that registry key deletions are also captured and stored in the 
.REG file. Please see the LLD doc by Charles Booher about the Registry spoofing 
database. 

R-FileCapture: The installation monitor records all the files created in the two 
kinds of directories: the install directory (the Fj group described above) and the 
common directories (the Fcsu group). All the files created are copied to the 
Installation Set and the File Relocation Map (FRM) created for the Fcsu group 
files. <Note: as far as a system common DLL is concerned, the eStream client 
should (a) overwrite the existing DLL if it exists (b) spoof it if doesn't exist. This 
is necessary because some installations may depend on newer versions of, say 
MSVCRT.DLL and in Windows there is no way to maintain different versions of 
the same DLL>. 

R-InitiaJProfiling: The Builder must be able to gather initial set of application 
profile data. This data consists of the page access pattern for starting and 
immediately shutting down an application. 

R-Packaging: The Builder must package the eStream Set into a easily 
manageable packages suitable for ASP administrators to download to their 
servers. The package can be divided into two sets: 

1 . Installation Set - an appInstallBlock which is a set of data needed to setup the 
client machine for running a particular eStream application. The 
appInstallBlock is converted into an installation executable for simplifying the 
initial application set-up on the client machine. 

2. Run-time Set - a set of files associated with a particular application. At run- 
time, appropriate pages from this set of files is streamed to the client. 

R-Merging (not supported in version 1.0): The Builder must be able to collect 
per-user profile data from the Profile Server and merge the profile data into a 
combined data usable for updating the profile data in the appInstallBlock. This 
profile data can also be collected for use by the ASP or application developers. 
R-NoQuietOperation: The Builder is not required to be run in an environment 
where no other applications are running. But, since the Builder operates by 
invoking application installation program, it inherits any restrictive "Quiet- 
Operation" requirement from the installation program. Thus, if the installation 
program of an application has a "Quiet-Operation" requirement, then the "Quiet- 
Operation" must be enforced by the user when running the Builder. 
R-AHCIient: The Builder should provide functionality to create installation set(s) 
for each of the clients eStream 1 .0 is going to support. <Preferably there should be 
only one builder program that should recognize the OS it is running on and should 
create the appropriate installation set. Also if possible, we should be able to "diff ' 
installation sets for different OSs and if they are same, we should be able to create 



a single installation set for those OSs. The clients to be supported are W2K, 
WinNT4.0 and Win98>. 

• R-AppIdGeneration: It should be possible to change the appld of the eStream set 
when an ASP wants to "install" the eStream set in order to host it. Typically the 
builder will generate a default appld number for a new application which can be 
overridden by the ASP installer by using a Builder tool 

• R-SuiteSupport: It should be possible to create a merged eStream set for a suite 
of applications. E.g. Office consisting of Word, Excel and Powerpoint. This could 
be done either by providing a tool for merging multiple eStream sets or by 
allowing the builder to serially monitor multiple installations in a session and then 
allowing the user to create a single package at the end of the session. 

• R-Testing: It should be possible to test the Builder using a stand-alone tester and 
not require the eStream client+server programs. 

• R-UpgradeSupport: The appInstallBlock should have support for indicating 
upgrades at the support site. E.g. When an eStream application is upgraded at the 
server (not as a separate app), the client will no longer be able to access/use it. We 
should provide some version of the appInstallBlock itself so that clients should 
detect that they will need to download the appInstallBlock again. 

• R-Manuallntervention: In the process of creating an eStream set it should be 
possible for the user to delete file entries and registry entries manually to "trim" 
the eStream set if she so desires assuming the user knows what she is doing. 

Issues 

• Profile Sequence Matrix is different for different machine configuration even if 
the user's usage pattern is the same. 

• Profile Sequence Matrix doesn't contain the right successor profile information as 
eStream cache is warmed up and pages from the cache is replaced. 

• Merging Module must take different machine configuration into account. Should 
this information be uploaded by the client as the same time it uploads the Profile 
Sequence Matrix to the Profile Server? 

• What is the difference between profiling based on the page sequencing seen by 
the eStream Cache Manager versus the page sequencing missed by the eStream 
Cache Manager? 
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Sub-components of the Builder covered here (note, these are logical divisions and not 
necessarily physical): 

1 ) Builder UI (the only program directly invoked by the user). 

2) Install Monitor module. 

3) File System and Registry filter driver (FSRFD). 

Interfaces: 

1) User to Builder UI: 

There are multiple Use Cases represented by this interface: 

1 . Monitor a new install: user needs to provide the (a) setup.exe path and (b) 
the destination drive where the app will be installed. If this is the second 
or subsequent install in the session then destination drive should be same 
as previous ones. After the Builder has finished running setup.exe the user 
needs to tell the Builder whether the installation was successful or not 
(unless the Builder can figure it out from the exit code of the setup.exe 
process). If the installation was not successful, Builder will erase all the 
installation set data created. 

Validation: Check that the destination drive is not the same as the system 
drive (e.g. C: on most systems), since the install monitor won't be able to 
differentiate between common files and installation files in that case. 

Note: The Builder user needs to start with a machine that is "pristine" i.e. 
this machine should only have the OS installed and no other application or 
data files. This way we get "maximal" installation to cover all the client 
configurations i.e. the application install will not be affected by any 
existing applications or registry settings etc. <At this point it is not clear if 
"pristine" OS includes any service packs>. 

2. Manual Edits: This allows the user to manually delete file entries and 
registry entries. Normally this should not be used. This functionality 
provides a screen that allows entry deletions for registry/files. 

3. Initial Profiling: This allows the initial profile to be created for the 
application(s) installed in this session. <User needs to provide the .EXE 
that will be run for the initial profile??>. 

4. Create eStream set: When the user selects this option, the builder creates 
an eStream set for all the installations done in the session. The Builder 



maintains a current appld counter in the registry (not to be confused with 
the registry monitoring we are doing), and uses that to prompt the user 
with the appld to be used for this app set. The user can override that appld. 
The user is also prompted for a version number of the applnstallBlock. 
The output eStream set is cre^ed^ith the version number. Note: The 
Builder will NOT compress or encrypt any of the application files. This is 
done by the eStream server prior to transmitting the files to the client. 

2) Builder UI to Install Monitor: Assuming the install monitor is a separate module, 
' there will be an entry function to invoke the install monitor thus: 

unsigned int InstallMonitor( 

IN PUNICODE^STRING setup name, /* setup.exe path */ 

IN PUNICODE_STRING dest_drive, /* dest drive for install */ 

IN PUNICODE_STRING filejbrjilejnfo, /* file for storing file info */ 

IN PUNICODE_STRING file Jbrjeljnfo, /* file for rel info */ 

IN PUNlCODE_STRING fileJbr_regjnfo, /* file for storing reg info */ 

IN BOOL append); /* append to above 2 files if they exist */ 

<The file formats for the file info, rel file info and the reg info are yet to 
be fixed>. 

3) Install Monitor to setup.exe of the application: The install monitor invokes the 
setup.exe using CreateProcess or similar Win32 API. Note that it has to invoke 
this as suspended, since we want to get the process ed and pass it on to the 
FSRFD so that the FSRFD can start monitoring all the requests for this process as 
well as its children. After that we want to resume the setup.exe process. 
(Alternatively, we can just pass the process id of the InstallMonitor process and 
the setup.exe would be the child of this process). 

4) Install Monitor to FSRFD: The interface between these two is going to be very 
similar to the installmon interface in the prototype. The interface to FSRFD will 
support the following: 

a) MON_ACTIVATE: Start monitoring. Will pass the process-id, destination 
drive, system drive 

DeviceIoControl(hDevice, // handle to our FSRFD device 
MON ACTIVATE, // activate control code 
activateOptions, // process-id, dest and sys drive 
sizeof(activateOptions), 
0,0, 

&numBytesRetumed, 

0); 



b) MONDEACTIVATE: Stop monitoring. 



DeviceIoControl(hDevice, 
MON DEACTIVATE, 
0,0, 
0,0, 

&numBytesReturned, 
0); 

c) We would use event objects (using IoCreateNotificationEvent as in the 
installmon program) for the FSRFD to signal the Install Monitor about the 
availability of new data. There would be a single event object with the path 
"\\BaseNamedObjects\installmon" that the FSRFD will use to signal the 
Install Monitor that a new reg entry or file entry is available. 

d) MON_GET_ENTR Y : Get the next entry from the FSRFD. This would be 
either a file or registry update. 

DeviceIoControl(m_deviceHandle, 

MON_GET_ENTRY, 
0,0, 

monitorEntry, 

sizeof(monitorEntry), 

&numBytesReturned, 

monitorEntry is a struct that is used to convey registry or file updates to the 
install monitor. The struct tentatively looks as follows: 

struct MonitorEntryt { 

UCHAR regOrFile; // 'R' for registry 'F' for file 
UCHAR addOrDelete; // 'A' for add, 'D' for delete, C U' for update 
UCHAR valueType; // only registry adds, value type 
ULONG nameLength; // length of name 
ULONG valueLength; // name of value (when it is a string) 
UNICODE_CHAR name[l]; //name of nameLength followed by value 

// valueLength 

}; 



Install Monitor Output contents 

ApplnstallBlock Contents 

• Aib Version: Magic number or appInstallBlock version number (which identifies 
the version of the appInstallBlock structure rather than the contents). 



Appld (this may be a 64-bit number where the low 32 bits identify app's version 
number). Appld for Word on Win98 will be different from Word on WinNT (if it 
turns out that Word binaries are different between NT and 98). 
VersionNo: Version number, (this allows us to inform the client that the 
appInstallBlock has changed for a particular appld). Note: this is different from 
the low 32 bits of the appld. E.g. Word 97 and Word 98 will be differentiated 
using the low 32 bits of the appld. Or versions 1 .0 and 2.0 of a software might be 
differentiated using the low 32 bits of the appld. However this field will be used 
when the Word 97 has been updated using a patch and the old binaries (for the 
same appld) are no longer available. 

ClientOSBitMap: Client OS supported bitmap or ED: for Win2K, Win98, WinNT 
and other future Oss we might support (it should be possible to say that this 
appInstallBlock is for more than one OS). 

<CIientOSServicePack: We might want to store the service pack level of the OS 
for which this appInstallBlock has been created. Note that when this field is set 
we cannot use multiple OS bits in the above field ClientOSBitMap>. 
ISM: Installation Set Map (ISM): contains 
o Fileld (-1 denotes a deleted file) 

o Full path of the file where the file resides in eStream (e.g. Z:\. . .) 
Note: a range of filelds will be reserved for eStream's own files. E.g. the 
appInstallBlock files could be referenced using these reserved filelds. 
FRM: File Relocation Map (FRM): contains 

o Fileld (same as the one in the ISM). 

o The path where the application expects it (e.g. C:\. ..) 
Note: The client should compare the FRM with the client machine. Any files 
that exist on the system (such as MSVCRT.DLL) will need to be copied to the 
client machine (instead of spoofing the file) and the client will need to be told 
to reboot the system. 
Registry spoof data: contains: 

HKEYCURRENTUSER 
Estream 

Registry Spoof 
Add 

HKCR 

HKLM 

HKCC 

Remove 

HKCR 

HKLM 



HKCC 



• Initial profile data: The Builder will profile the application usage to get the 
sequences of the blocks accessed. The Builder will gather the access sequences 
sent by the OS to the file system. The profiler will stop when the profile noise 
exceed some tolerance level (ie. If X percentage of the pages of the application 
being profiled has been kicked out of memory). A list of entries where each entry 
has the following format: 

o previous fileDD 

o previous blockED (blockID may be replaced with Offset and Length if 

variable sized cache blocks is supported) 
o next filelD 

o next blockID (blockID may be replaced with Offset and Length if variable 

sized cache blocks is supported) 
o frequency. 

• Initial pages for the app - The size of the initial prefetch blocks will be 
determined based on the minimum size of the data required to achieve good 
compression. A list of entries where each entry has the following format: 

o FilelD 

o BlockID (blockID may be replaced with Offset and Length if variable 
sized cache blocks is supported). 

• A comment field - this text might be used by the client to show the eStream app 
user any relevant info 

• Special processing needed for this app - this could be a pointer to an EXE/DLL or 
the code itself in the appInstallBlock or a batch file. 

Other Tools Related to the Builder 



AppInstallBlock Editor/Modifier 

This tool allows you to edit/modify one or all of : 

a) the appld 

b) the appInstallBlock version number 

c) OS bit map (since we may need to add bits for other OSs once we know that 
binaries are the same for them). 

of the appInstallBlock portion of the eStream set created by the builder. 

Applnstall Compare Tool 

This tool allows one to compare two eStream sets (both the appInstallBlocks and the 
actual files contained in the file set). Any differences are flagged to the user. This will be 
useful when we know that an application has the same set of binaries for multiple client 

OSs. 

This tool will also be used to create appUpgradeBlock that will be used by the eStream 
client to seamlessly upgrade client's apps without necessarily destroying their config 
files. This is the only way to do it without needing an uninstall of the previous version. 



Builder Tester 

This program allows us to test the output of the Builder without requiring the eStrearn 
client/server set up. 
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Functionality 

The eStream Application Builder Package Manager is responsible for packaging data 
gathered from the Installation Monitor, the Profile Manager, and the Upgrade Monitor 
into a set of data called the eStream Set. For the detail format of the eStream Set, see the 
separate document on eStream Set. 

The Package Manager must perform the following task: 

□ Create the applnstallBlock containing C-File and Registry data from the Install 
Monitor; Prefetch data from the Profile Manager; and Updated C-File and Up- 
dated Registry data from the Upgrade Monitor 

□ Create a custom installation DLL needed by a specific applications and add to the 
applnstallBlock 

q Create directory files associated with each directory of the application director 
and add metadata to the directory 

□ Create directory files associated with each Windows directory containing both the 
Spoofed files and Z-files 

□ Create Concatenated Application File (CAF) which is just a juxtaposition of the 
application files, eStream directory files, and ApplnstallBlock 

□ Create Size Offset File Table (SOFT) which is a mapping of fileNumber to offset 
of the start of the CAF file 

□ Create Root Version Table (RVT) which is a mapping from the version of root to 
the file number of the root directory file 

□ Archive the CAF, SOFT, and RVT into a single structure called eStream Set suit- 
able for uploading to the eStream Servers. 

Data type definitions 

The Package Manager doesn't have any internal data types. It must accept and under- 
stand data structures received from the Install Monitor and the Profile Manager. See In- 
stall Monitor and Profile Manager components for the description of the data structures. 

The Install Monitor is responsible for generating the following list of information: list of 
copied-files, list of spoof-files, list of files with file numbers, list of add registry entries, 
and list of delete registry entries. The list of copied-files contains the files copied into 
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non-application specific directories. The list of spoof-files consists of the files too large 
to be downloaded to the client in the AppInstallBlock. Those files are copied into some 
special directory on the Z drive for streaming. The list of files with file numbers consists 
of the files copied into the standard "Program Files" directory and the files that will be 
spoofed. The registry information is a list of registry key added or removed during the 
installation of the application. 

Struct FilelndexTable { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING FilePathName; 
ULONG FileNumber; 
} Entries [NumEntries] ; 

}; 

Struct FileCopied { 
UINT NumEntries; 
Struct Entry { - 

PUNICODE_STRING FilePathName; 
} Entries [NumEntries] ; 

} 

Struct FileSpoofed { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING OldFilePathName ; 

PUNICODEJSTRING NewFilePathName ; 
} Entries [NumEntries] ; 

}; 

Struct Registrylnfo { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING KeyName ; 
PUNICODE_STRING ValueName; 
PVALUEJDATA ValueData; 
} Entries [NumEntries] ; 

}; 

Struct Inilnfo { 
UINT NumFiles; 
Struct FileEntry { 

PUNICODE_STRING FilePathName; 
UINT NumSections ; 
Struct SectionEntry { 

PUNICODE_STRING SectionName; 
UINT NumValues ; 
Struct Entry { 

PUNICODE_STRING ValueName; 
PVALUEJDATA ValueData; 
} Entries [NumValues] ; 
} Entries [NumSections] ; 
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} Entries [NumFiles] ; 

}; 

The Profile Manager generates AccessCounts and the PrefetchBlocks data with the struc- 
tures shown below. 

Struct AccessCounts { 
UINT NumEntries ; 
Struct Entry { 

PUNICODE_STRING FilePathName; 
ULONG Frequency; 
} Entries [NumEntries] ; 

}; 

Struct PrefetchBlocks { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING FilePathName; 

ULONG BlockNumber; 
} Entries [NumEntries] ; 

}; 

The eStream Set has the following data structure (described in more detail in the separate 
eStream Set document): 

Struct eStreamSet { 

Struct eStreamSetHeader header ; 
Struct eStreamSetRVT rvt; 
Struct eSt reamSet SOFT soft; 
Struct eStreamSetCAF caf; 

}; 

Interface definitions 
Function 1 : CreateEStreamSet 

// Create the initial eStream Set from the data 
// retrieved from the Install Monitor and the 
// Profile Manager. 

// This function is called only by the Builder 

// UI after data is obtained from Install 

// Monitor and Profile Manager. 

int CreateEStreamSet ( 

IN PFILE_INDEX_TABLE FIT, 

IN PFILE_SPOOFED Spoof Files, 

IN PFILE_COPIED CopiedFiles, 

IN PREG I S TRY__ INFO AddRegistry, 

IN PREGISTRY^INFO RemoveRegistry , 

IN PINI_INFO Inilnfo, 

IN PACCESS_COUNTS AccessCounts, 

IN PPREFETCH_B LOCKS PrefetchBlocks, 
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IN PVOID DllCode, 

IN PUNICODE_STRING Comment, 

OUT PESTREAM__SET EstreamSet) 

Input : 

FIT: File Index Tree contains the file 
number of the directories, spoofed 
files, and standard files 

CopiedFiles: pointer to a list of files 
To be copied to AppInstallBlock 

Spoof Files: pointer to a list of files 
To be spoofed on the client 

AddRegistry: pointer to a list of registry- 
Data to add 

RemoveRegistry : pointer to a list of 
Registry data to remove 

Inilnfo: pointer to a list of ini changes 

AccessCounts : pointer to the list of 
Fil es with the access frequency 

Pref etchBlocks : pointer to the prefetch data 
To be inserted into the appInstallBlock 
Of the eStream Set 

DllCode: pointer to DLL Code 

Comment : pointer to comment string 

Output : 

EstreamSet : pointer to the eStream Set 

Return Value: 

Success or failure of the packaging process 

Comments : 

The eStream Set will be large for most 
application. Intermediate data will be 
stored on the local hard-drive. 

Errors : 

OutOf Storage : failure to find enough storage 
For this eStream Set 
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FileNotFound: failure to find the files 
Specified by either ListCFiles or 
ListZFiles 

Function 2 : UpgradeEStreamSet 

// Upgrade the eStream Set to the latest 

// version. This function is only called by 

// the Upgrade Manager within the same process. 

int UpgradeEStreamSet ( 

INOUT PESTREAM_SET EstreamSet, 

IN PFILE_INDEXJTABLE UpgFIT, 

IN PFILEJSPOOFED UpgSpoof Files , 

IN PFILE_COPIED UpgCopiedFiles , 

IN PREGI STRY_INFO UpgAddRegistry , 

IN PREGI STRY_INFO UpgRemoveRegistry, 

IN PACCESSJZOUNTS UpgAccessCounts , 

IN PPREFETCHJ3LOCKS UpgPref etchBlocks , 

IN PVOID UpgDllCode, 

IN PUNICODE_STRING UpgComment) 

Input : 

UpgFIT: File Index Tree contains the file 
number of the directories, spoofed 
files, and standard files 

UpgCopiedFiles: pointer to a list of files 
To be copied to AppInstallBlock 

UpgSpoof Files: pointer to a list of files 
To be spoofed on the client 

UpgAddRegistry: pointer to a list of 
Registry data to add 

UpgRemoveRegistry: pointer to a list of 
Registry data to remove 

UpgAccessCounts: pointer to the list of 
Files with the access frequency 

UpgPref etchBlocks : pointer to the prefetch 
Data to be inserted into the 
AppInstallBlock Of the eStream Set 

UpgDllCode: pointer to DLL Code 
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UpgComment : pointer to comment string 
Output : 

EstreamSet : pointer to the eStream Set 

Return Value: 

Success or failure of the packaging process 

Comments : 

The eStream Set will be large for most 
application. Intermediate data will be 
stored on the local hard-drive. 

Errors : 

OutOf Storage : failure to find enough storage 
For this eStream Set 

FileNotFound: failure to find the files 
Specified by either ListCFiles or 
ListZFiles 

Function 3 : InsertProfileData 

// Insert profile and prefetch data into the 
// eStream Set. This function is only called by 
// the Merge Manager within the same process. 

int InsertProfileData ( 

INOUT PESTREAMJSET EstreamSet, 

IN PACCESS_COUNTS AccessCounts , 

IN PPREFETCH_BLOCKS Pref etchBlocks) 

Input : 

EstreamSet: pointer to old eStream Set 
Before the insertion of the profile 
Data 

AccessCounts: pointer to the list of 
Files with the access frequency 

Pref etchBlocks: pointer to the prefetch data 
To be inserted into the appInstallBlock 
Of the eStream Set 

Output : 

EstreamSet: pointer to the new eStream Set 
Return Value: 
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Success or failure of the insertion process 
Comments : 

The eStream Set will be large for most 
application. Intermediate data will be 
stored on the local hard-drive. 

Errors : 

OutOf Storage : failure to find enough storage 
For this eStream Set 

FileNot Found: failure to find the files 
Associated with the prefetch blocks 

Component design 

The pseudo-code for the function CreateEStreamSet is described below: 
{ 

Create AppInstallBlock (ABB) from the following input files: 
o SpoofFiles 
o CopiedFiles 
o AddRegistry 
o RemoveRegistry 
o Prefetch 
o Comment 
o DLLcode 

Assign AppInstallBlock with a unique fileNumber given by the IM; 
Record Root fileNumber in the first entry of Root fileNumber Table (RFT); 
Move AppInstallBlock under the Root directory by adding a new entry in the 

Directory structure; 
Create a Concatenation Application File (CAF) header; 
Create a Size Offset File Table (SOFT) header; 
For each (file in FIT) { 

If (file is a directory) { 

Create the directory with new list of fileNumber, filename, and 
Metadata; 

} Else { 

Find the file in the proper location on the HD; 

} 

Append the file or directory to the end of the CAF file; 
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Append the fileNumber, offset into CAF, and size of file in SOFT; 

} 

Archive CAF, SOFT, and RFT into a single eStream Set; 
Return eStream Set; 

} 

The pseudo-code for the function UpgradeEStreamSet is mentioned below: 
{ 

Extract previous version PrevAppInstallBIock from eStream Set; 
Create new AppInstallBlock with new FileNumber; 

Extract PrevSpoofFiles and PrevCopiedFiles from PrevAppInstallBIock; 
Divide the C-Files into SpoofFiles and CopiedFiles; 
Add PrevSpoofFiles to SpoofFiles; 
Add PrevCopiedFiles to CopiedFiles; 

Extract PrevAddRegistry and PrevRemoveRegistry data from 

PrevAppInstallBIock; 
Add any unique ((UpgAddRegistry plus PrevAddRegistry) minus 

UpgRemoveRegistry) in the new AppInstallBlock AddRegistry section; 
Add any unique ((UpgRemoveRegistry plus PrevRemoveRegistry) minus 

UpgAddRegistry in the new AppInstallBlock; 

Add UpgPrefetch data to new AppInstallBlock; 
Add UpgDllCode data to new AppInstallBlock; 
Add UpgComment data to new AppInstallBlock; 

For each (directory in UpgFIT) { 

If (any child fileNumber has changed) { 

Create new directory with updated fileNumber; 

Append file to end of Concatination Application File (CAF); 

Append Size Offset File Table (SOFT) with new entry; 

} 

} 

Append new AppInstallBlock to the end of CAF file; 
Prepend Root FileNumber Table (RFT) with new Root entry; 
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Archive CAF, SOFT, and RFT into a single eStream Set; 
Return eStream Set; 

} 

The pseudo-code for the function InsertProfileData is mentioned below: 
{ 

// not needed unless merging of uploaded profile data is supported 

} 

Testing design 

This document must have a discussion of how the component is to be tested, 
o Unit testing plans 

The plan for unit testing Package Manager includes the development of a driver 
program. This driver interfaces to the Package Manager and invokes the func- 
tions with different parameters. The list of possible cases is described below: 

1 . Test all interfaces by driving the input parameters with different type of add 
and remove registry values. 

2. Test all interfaces by driving the input parameters by varying numbers of 
spoof and copied files. 

3. Test all interfaces by driving the input parameters with some prefetch infor- 
mation. 

4. Test all interfaces for meaningless input values from the IM and PM. 
o Prefetch block containing file number not assigned by IM. 

o IM assigning non-continguous file numbers. 

5. Test upgrade interface for capability to detect and handle bad eStream Set 
gracefully. 

6. Test upgrade interface and make sure it can detect overlapping file number as- 
signments. 

7. Test upgrade interface and make sure prefetch blocks are not referencing old 
file number from previous versions. 

o Stress testing plans 



o Coverage testing plans 



o Cross-component testing plans 

The output data from the Package Manager is called the eStream Set. This 
eStream Set is the input to a stand-alone test program called the eStream Extrac- 
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tor. The Extractor unpacks and 'install' the eStream Set into the local machine 
without an eStream client file system installed. This test is used to quickly verify 
that the eStream Set can be run on a pristine machine. Some of the possible varia- 
tions of the Extractor test includes: 

1 . Non-default system variable names. I.e. %SystemRoot% set to "D:\Win" in- 
stead of"C:\Winnt'\ 

2. Non-default eStream FS drive letter. Use Y instead of Z. 

Upgrading/Supportability/Deployment design 

The Package Manager logs all error messages to a predefined file common to all compo- 
nents of the Builder program. Every Builder component prints the error message along 
with its component name. This allows the user of the Builder program to quickly track 
down any problem during the Building of a new eStream Set. 



Open Issues 

o Which Builder component creates the installation DLL when the application 

needs the custom installation code? Is a new component needed to create the cus- 
tom DLL separately and insert into AppInstallBlock in the eStream Set as 
needed? 
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Functionality 

The eStream Application Builder Profile Manager is responsible for the following: 

□ Receive request from the Ul Component for one or more application executables 
to profile. 

□ Accumulate each run of the profile data in a data structure suitable for merging. 

Q Invoke each application executable for a fixed amount of time, for a fixed number 
of prefetch blocks, for a simple start-stop of the program, or for multi-level profil- 
ing based on scripts or manual usage of an application. 

□ Communicate with File Access Monitor (FAM) kernel-mode driver using 
lOCTLs to start and stop profiling. 

o Obtain the complete file access sequence data from the FAM. 

□ Process the file access sequence into two parts: a table of file access frequency 
and a list of prefetch blocks. 

□ Send the resulting profile data to Package Manager component for integration into 
the AppInstallBlock. 

This component will probably exist as a class object and will be instantiated by the 
Builder user interface component. The component will run in the same process as the 
user interface component. Please see Builder User Interface component document for 
more information on that component. 



Data type definitions 

The Profile Manager imports the file access sequence from the FAM. The data structure 
is described below: (Please see the File Access Monitor for detail information on the 
meaning of each field in the data structure) 



Struct SequenceData 
{ 

UINT NumEntries; 
Struct Entry 

{ 

UNICODE_STRING FilePathName ; 
BOOL IsAccessingMetaData,- 
ULONG Offset; 
ULONG Size; 
} Entries [NumEntries] ; 
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}; 

Profile Manager creates two data structure from the data received from the Install Moni- 
tor and the FAM. The consumer of the output data structure is the Package Manager. 
These data structures is described in the following two structures: 

Struct PrefetchBlockList { 
UINT NumSections; 
Struct Pref etchBlocks { 
UINT BlockType; 
UINT NumEntries; 
Struct Entry { 

UNICODE_STRING FilePathName ; 
ULONG BlockNumber; 
} Entries [NumEntries] ; 
} Entries [NumSections] ; 

}; 

Struct Prof ileAppli cat ions { 
UINT NumEntries ; 
Struct Entry { 

UNICODE_STRING FilePathName; 
UNICODE_STRING Arguments; 
} Entries [NumEntries] ; 

}; 

The access count is used to order the list of files in a directory according to metadata file 
access frequency. The prefetch data is encorporated into the AppInstallBlock by the 
Package Manager to be used by the eStream Client. 

Interface definitions 
Function 1 : StartProfiling 

int StartProfiling ( 

IN PPROFILE_APPLICATIONS AppList, 

IN UINT Type, 

IN UINT TypeData, 

OUT PPREFETCH_BLOCKS Pref etchBlocks) 
Input : 

AppList: a list of file pathnames and 
Arguments to run the application. 

Type: type of profiling to do 

SIMPLE: start and stop application 
TIMEBASED: profile for a fixed time and 
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terminate application 
SIZEBASED: profile for a fixed size of 
Profile data 

TypeData: extra data for profile type 
If Prof ileType==SIMPLE, TypeData is 
Ignored 

If Prof ileType==TIMEBASED, TypeData is 

Length of time in seconds 
If ProfileType==SIZEBASED, TypeData is 

Size of profile data in bytes 
If Pro f i 1 eType = = COMMANDBAS ED , TypeData is 

Pointer to a possible list of script 

Files to be invoked 

Pref etchBlocks : List of prefetch blocks 
To add to the AppInstallBlock 

Return value: 

Success or failure code of the profiling 

Comments : 

The profile manager component actually send 
IOCTLs to the file filter device driver to 
Start and stop the gathering of the profile 
Data and to retrieve the profile data. 

Errors : 

FileNot Found: some of the application 

Executables in the list may not exist or 

Not readable 
Prof ileTimeout : failed to gather the desired 

Size of the profile data after certain 

Amount of time 
DriverFailure : File Access Monitor return 

Failure code to the Profile Manager which 

Must propagate the error to the user 

Interface 

Component design 

Application Invocation 

To start profiling, the component must have a list of application pathname to be invoked. 
The pathname may be an executable with a list of arguments. Or the pathname may be a 
Windows short-cut file. If the pathname is an exe file, then the standard Win32 API Cre- 
ateProcessQ is used. On the other hand, if the file is a Windows short-cut file, then the 
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component needs to extract relevant information from the short-cut file to make the 
proper CreateProcessQ invocation. The following is a pseudo-code for extracting path- 
name and argument information from the short-cut file using IShellLink interface: 

Getlnf oFromShortCutFile (char *strPath, char *strArg) 

IshellLink *psl; 
WIN32_FIND_DATA fd; 

if (SUCCEEDED ( CoCreatelstance (CLSID_ShellLink, 
NULL, 

CLSCTX_INPROC_SERVER , 
IID_IShellLink, 
(LPVOID*) &psl) ) ) 

{ 

psl->GetPath(strPath, MAX_LEN , &fd, 0) ; 
Psl->Get Arguments (strArg, MAX_LEN) ; 
psl->Release () ; 

} 

} 

Command-based Profiling 

One of the options for profiling include the ability to identify blocks of files accessed 
when specific application command is invoked. The profiler prompts the user for the de- 
sired actions (ie. Open document, save document, etc) and gathers file blocks that are ac- 
cessed corresponding to those actions. These commands are saved into the Applnstall- 
Block for eStream client to intelligently pick the proper set of blocks to stream. The fol- 
lowing is an enumeration of some divisions of prefetch blocks: 

o Start Application 

o End Application 

o Save Document 

o Open Document 

o Cut Sections 

o Copy Sections 

o Paste Sections 



Communication with kernel-mode driver (FAM) 

The Profile Manager communicates with the kernel-mode driver to retrieve the informa- 
tion on the blocks of files accessed. The profile manager waits for a signal from the 
FAM indicating a new data is available for retrieval. FAM also signals the profile man- 
ager when the profiled application terminates. FAM uses KeSetEventQ to send a 4 data 
available' event signal to the profile manager. Profile manager calls KeWait- 
ForSingleEventQ or KeWaitForMultipleEventQ to wait for a signal from the kemal-mode 
driver. KeCleorEvenlQ is called by the FAM when the signal to profile manager should 
be deactivated. 
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Pseudo-code 

The pseudo-code for the function StartProfiling is described below: 
{ 

Initialize GlobalFileAccessCounts and GlobalPrefetchBlocks; 

Load FAM if not loaded; 

For each (executable in the list of application) { 

Start FAM by sending it process ID of the Profile Manager; 

Create new process for this executable and run it; 

Switch (Type of Profiling) { 

Case SIMPLE: 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} Until application start up; 
Break; 
Case TIMEBASED: 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} Until fixed time unit; 
Break; 
Case S1ZEBASED: 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} while (size < fixed amount); 
Break; 

Case COMMANDBASED: 

For each command in the list { 
If (script exist) 

Run script on the executable program; 
Else 

Prompt operator for proper action; 
Omnishift Technologies, Inc. 5 Company Confidential 
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Loop { 

Wait for an event from FAM; 
Get Status from FAM; 
Get SequenceData from FAM; 
} Until script completed; 

} 

} 

Send WMQUIT message to the application process; 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} Until application quit; 
Inform FAM to stop gathering profile data; 

Compute PrefetchBlocks from the SequenceData and append to 
GlobalPrefetchBIocks; 

} 

Unload File Access Monitor (if possible); 
Return GlobalPrefetchBIocks; 

} 

Testing design 

o Unit testing plans 

The plan for unit testing the Profile Manager includes developing a driver to con- 
nect to the interface between the Profile Manager and the Builder UI. The driver 
conducts the following types of tests: 

1 . Test different type of profiling including simple profiling, time-based profil- 
ing, size-based profiling, and script-based profiling. 

2. Test different executable programs and make sure the output data "makes 
sense". 

3. Test a list of executables for merging capability of the Profile Manager. 

4. Test the interface between Profile Manager (PM) and the File Access Monitor 
(FAM) using FAM as the test driver. The FAM can check for any valid 
IOCTL calls from the PM. FAM can also reply to 10CTL calls with different 
values in the IRP to simulate all possible cases. 
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o Stress testing plans 



o Coverage testing plans 



o Cross-component testing plans 

Cross-component testing for the Builder program is described in the Package 
Manager low-level design document. 

Upgrading/Supportability/Deployment design 

The Profile Manager logs all error messages to a predefined file common to all compo- 
nents of the Builder program. Every Builder component prints the error message along 
with its component name. This allows the user of the Builder program to quickly track 
down any problem during the Building of a new eStream Set. 

Open Issues 

o How to automate profiling so the application doesn't require any user interven- 
tion? Look into using Rational TestSuite programs. 

o Can a subset of Winstone be used for profiling? How do we determine which part 
of the profile data is more useful to the end-user? 

o Should Profile Manager actually create data structures like PrefetchBlocks which 
require FileNumber assignments? Or should Profile Manager just create the out- 
put data without knowning FileNumbers? Then Package Manager associates the 
file numbers assigned by the Install Monitor with the profile data gathered by the 
Profiler. 
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Tricky Builder Issues 



Author: Sanjay Pujare 

This document enumerates all those tricky issues that may make the Builder's job 
difficult. Even though some solution may be proposed for some issues, not every issue 
would have a solution described in this document. The purpose of this document is 
mainly to keep track of Builder issues that may impose some limitations on the eStream 
technology. This way Omnishift marketing and deployment are aware of these 
limitations. 

1) The Builder cannot capture updates to existing files in an intelligent fashion (i.e. 
if the updates are based on a context or existing contents, it is very difficult to 
capture that). So the current Builder will just flag an error, if such an update 
occurs. 

Solution 

□ These updates are probably very rare, so we can defer it to the next 
release. 

□ For this release, we can try to solve this on a case by case basis e.g. we 
will try to solve this issue for INI files. 

□ Based on our understanding of general app installations, we might be able 
to make some generalizations that we can use in eStream e.g. Only certain 
files get updated; there is a definite pattern of updates. 

2) The current Builder drivers are based on the NT driver model, and hopefully we 
can implement the same functionality in the Win98 drivers, but this needs to be 
ensured. (This shouldn't be an issue, but...) 

3) We need to think some more about those cases when device drivers are installed 
by apps. Issues that can arise: 

□ This may not work correctly on the eStream client, just because the driver 
installation didn't take place properly. 

□ The Builder would need to be able to figure out in an automated way if a 
client reboot is required or not. 

□ If the driver installation is h/w or s/w specific that can be difficult to tackle. 
Solution 

□ As we eStream more apps and gain more experience, we should be able to 
figure out solutions. 

4) There could be an ambiguity when the Installmon is trying to change absolute 
paths (or absolute values in general) to relative paths, e.g. A path like C:\WINNT 
can be changed to %SystemRoot% or %windir% since both of those environment 
variables are set to "C:\WINNT" on my system. 

Solution 

□ We can prioritize env vars and registry keys as described in the BuilderUI- 
LLD design document. 



□ We should encourage Builder operators to use as distinct values as 
possible for env variables and registry keys for the Builder machine. 



eStream BuilderUI Low Level Design 

Sanjay M Pujare 
<Date> 

Functionality 

The BuilderUI is the user interface part of the Builder. The operator uses this interface to 
use various functions provided by the Builder. Note that this Ul may or may not be a 
graphical user interface. This low-level design is based on the assumption that a graphical 
user interface is not necessary. 

Data type definitions 
Interface definitions 
Component Design 

When the Builder UI is invoked with command line arguments which indicate that this 
was invoked by the Runonce mechanism of Windows, the control is transferred to the 
InstallMon: : startCaptureAf terReboot() function with the command line 
arguments passed as arguments to the fimction. When the Builder is invoked normally, it 
presents a menu which is managed by the function MainMenu. 

MainMenu 

This function manages the following menu hierarchy. Each 
menu option (leaf node) is followed by a function name in 
parentheses that is called to handle the option. 

1) eStream Set Menu 

1) New eStream Set (NewEStreamSet ) 

2) Open eStream Set (OpenEStreamSet ) 

3) Save New/Upgraded eStream Set (EstreamSetCreation) 

2) Monitoring Menu 

1) Start Monitor (StartMonitor) 

2) Stop Monitor (StopMonitor) 

3) Check Status (CheckStatus) 

4) Inform Machine Reboot (Inf ormMachineReboot ) 

5) Get and Resolve Registry Set (GetRegistrySet ) 

6) Get and Resolve Files Set (GetFilesSet) 

3) Profiling Menu 

1) Set the location of app executable (GetAppPath) 

2) Gather Initial Profile (Gatherlni tialProf ile) 

4) eStream Set Creation Menu 

1) Set custom DLL (GetCustomDLL) 

2) Set User Comment (GetUserComment ) 

3) Set environment variables (GetEnvVars) 
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4) Set Reboot flag (GetRebootFlag) . 

5) Set License Agreement (GetLicenseAgreement) . 

NewEStreamSet 

{ 

If there is an existing eStrearn set that hasn't been 
saved, warn the user. 

Get the following values from the user: 

• Name of the app setup program in gAppSetup 

• Dest directory where app will be installed in gDest- 
Dir (provide a default value) . 

• Dest location to store the new eStream set in 
gDestEstreamPath (provide a default value) . 

} 

OpenEStreamSet 

{ 

If there is an existing eStream set that hasn't been 
saved, warn the user. 

Get the following values from the user: 

• Location of the existing eStream set in gSrcEstream- 
Path (provide a default value) _ 

• Whether the user wants to create an upgrade from this, 
or just wants to change the existing eStream set (gUp- 
grade) 

• If this is an upgrade (gUpgrade is true) , get all the 
values obtained by NewEstreamSet (i.e. gAppSetup, 
gDestDir, gDestEstreamPath) . 

Read the eStream set pointed to by gSrcEstreamPath; 
Load the existing file tables in gSrcCopiedFiles , 
gSrcSpoofedFiles and gSrcEFSFiles arrays; 

} 

EstreamSetCreation 

{ 

If there is no working eStream set, give error and re- 
turn. 

if (gUpgrade) { 

Call UpgradeEStreamSet () with appropriate arguments ; 

else { 

Call CreateEStreamSet () with appropriate arguments; 

if (gDestEstreamPath is not set) { 

assert (this is an update of an existing eStream set 
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and not an upgrade or a new eStream set creation); 
^ gDestEstreamPath = gSrcEstreamPath; 

Save the eStream Set from memory to file to 
gDestEstreamPath ; 

} 



StartMonitor 

{ 

If there is no working eStream set, give error and re- 
turn . 

if (gUpgrade) { 

Combine the gSrcSpoof edFiles and gSrcEFSFiles into an 
array f ileTableArray as expected by startCapture 
below; 

} 

Call InstallMon: : startCapture (gAppSetup, gDestDir, 
gUpgrade, f ileTableArray) ; 



StopMonitor 

{ 

If monitoring wasn't started, give error and return. 
Call InstallMon: rstopCapture () ; 



CheckStatus 

{ 

Ensure that monitoring was started and not stopped 
Call InstallMon: : checkSetupStatus () ; 



InformMachineReboot 

{ 

Ensure that we are in the middle of monitoring. 
Call InstallMon: :machineToBeRebooted () ; 



GetRegistrvSet 

{ 

Call InstallMon: :getRegistryList () ; 

Store the set in gNewRegistry data structure; 



GetFilesSet 

{ 

Call InstallMon: : getFilesList () ; 
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Store the set in gNewFiles data structure - 
Also we need to capture changes made to INI files; since 
this has not been taken care of in the app install block 
and other parts of the Builder+Client we will need to 
make. changes in all those components which are affected 



GetAppPath 

{ 

Ensure that all the InstallMon related data was captured 
Get the location of the app that needs to be run 
to gather profiling info; 
Store it in gProf ileAppPath; 

GatherlnitialProfile 

{ 

// this is the function that is used to get the initial 
// profile data (i.e. set of pages prefetched when an 
// eStream app is started for the first time) 
^ // implementation of this is yet to be defined 

GetCustomDLL 

{ 

Get the location of the custom DLL file, validate that it 
is a DLL and store the path in gCustomDLLPath; 

GetUserComment 

{ 

Get the user comment (optionally by browsing a text file) 
and store it in gUserComment ; 

GetEnvVars 

{ 

Call the InstallMon: : set EnvVars () function; 
GetRebootFlag 

Until we come up with an algorithm to determine if a re- 
boot is required for an eStream app, get this value from 
the user. The default is FALSE : we do not want to reboot 
the client PC when the user subscribes to this eStream 



Omnishift Technologies, Inc. 



4 



Company Confidential 



eStream <COMPONENT> Low Level Design 



} 

GetLicenseAqreement 

{ 

Get license agreement from the user. This could be: 

□ either, a default OmniShift license agreement 

□ or, a default ASP agreement 

□ or, license agreement that the app displayed. 

Let the user decide and enter the proper one. Provide a 
default based on our policy. 

} 

Interesting issues to deal with: 

Testing design 
Unit testing plans 

Testing of the UI itself is a comparatively trivial task. The testing will basically consist of 
traversing the whole menu hierarchy. Since the menu is similar to a typical File Open -> 
File Edit -> File Save kind of a user application, this can be tested using simple hooks. 

Stress testing plans 

Since the Builder will be used in house at least initially only simple stress testing should 
be necessary. Make sure that Builder doesn't crash in the middle of processing so that we 
don't lose important data. Performance is not considered to be important. 

Coverage testing plans 

Basically the following 3 paths will be exercised: 

1 . Create a new eStream set 

2. Open an existing eStream set to modify some data in it 

3. Open an existing eStream set to create an upgrade for it 

Cross-component testing plans 

Will be tested as a component of the whole builder. 

Upgrading/Supportability/Deployment design 

Deployment: This will be used in-house, so no deployment considerations. 
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Open Issues 

□ We need to think about the problem of converting absolute file paths discovered 
in the monitoring process to paths relative to some application or system registry 
key. Although most cases may not present a problem, we may have some difficult 
cases, which may make this problem non-automatable i.e. we may need some user 
intervention. Consider the case: 

KEYONE = C:\FOO\BAR 

KEYTWO = C:\FOO 

KEYTHREE = BAR 

If we notice that a file was copied to C:\FOO\BAR it won't be possible to convert 
this absolute path to a unique relative path since there are 2 solutions possible- 
%KEYONE% or %KEYTWO%\%KEYTHREE%. 

The way to solve this is by tracking only a set of well-known environment vari- 
ables and registry keys. Also in this set we prioritize all of them. So in the above 
case, %KEYONE% will be preferred over %KEYONE%\%KEYTHREE% just 
because of the way they were prioritized. 



Omnishift Technologies, Inc. 



6 



Company Confidential 



Format of the eStream Set 



CAF 



Header 



Root Version Table section 



Size Offset File Table section 



Regular File Data 



Root Directory 



ApplnstallBlock 



Directory 



Icon File Data 



Patched File Data 



Patched Directory 



Patched Root Directory 



ApplnstallBlock 



Header 



File Section 



Add Variable Section 



Remove Variable Section 



Prefetch Section 



Profile Section 



Comment Section 



Code Section 



License Section 



Root Version Table (RVT) 



Num entries 


Version 
Number 


Root File 
Number 


Version 
Name 


Metadata 




Version 
Number 


Root File 
Number 


Version 
Name 


Metadata 




Size Offset File Table (SOFT) 




eStream Directory 



Num entries 


Offset 


Size 


Offset 


Size 


Offset 


Size 


Offset 


Size 




Offset 


Size 


Offset 


Size 


Offset 


Size 4 



Header 


Metadata 


String Table 
Offset 


Name Len 


Hash 


File ID 




Metadata 


String Table 
Offset 


Name Len 


Hash 


File ID 




String 


Table 







v0.2 



eStream Set Format Low Level Design 



Sanjay Pujare and David Lin 
Version 03 



Functionality 

The eStream Set is a data set associated with an application suitable for streaming over 
the network. The eStream Set is generated by the eStream Builder program. This pro- 
gram converts locally installable applications into the eStream Set. This document de- 
scribes the format of the eStream Set. 

Note: Fields greater than a single byte is stored in little-endian format. All strings are in 
Unicode unless specifically stated otherwise. The eStream Set file size is limited to 2 A 64 
bytes. 

Data type definitions 

The format of the eStream Set consists of 4 sections: header, Root Version Table (RVT), 
Size Offset File Table (SOFT), and Concatenation Application File (CAF) sections. 

1. Header section 

o MagicNumber [4 bytes]: Magic number identifying the file content with 
the eStream Set 

o ESSVersion (4 bytesj: Version number of the eStream Set format, 
o AppID [16 bytes]: A unique application ID for this application. This 

field must match the AppID located in the AppInstallBlock. Guidgen is 

used to create this identifier. 

o RVToffset [8 bytes]: Byte offset into the start of the RVT section. 

o RVTsize [8 bytes]: Byte size of the RVT section. 

o SOFToffset [8 bytes]: Byte offset into the start of the SOFT section. 

o SOFTsize |8 bytes]: Byte size of the SOFT section. 

o CAFoffset |8 bytes]: Byte ofTset into the start of the CAF section. 

o CAFsize [8 bytes]: Byte size of the CAF section. 

o VendorNameLength [2 bytes]: Byte length of the vendor name. 

o VendorName [X bytes]: Name of the software vendor who created this 

application. I.e. "Microsoft". Null-terminated, 
o AppBaseNameLength (2 bytes]: Byte length of the application base 

name. 

o AppBaseName |X bytesj: Base name of the application. I.e. "Word 
2000". Null-terminated. 
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o MessageLength |2 bytes]: Byte length of the message text, 
o Message [X bytes): Message text. Null-terminated. 

2. Root Version Table (RVT) section 

The Root version entries are ordered in a decreasing value according to their file 
numbers. The Builder generates unique file numbers within each eStream Set in a 
monotonically increasing value. So larger root file number implies later versions 
of the same application. The latest root version is located at the top of the section 
to allow the eStream Server easy access to the data associated with the latest root 
version. 

o NumberEn tries |4 bytes] : Number of patch versions contained in this 
eStream Set. The number indicates the number of entries in the Root Ver- 
sion Table (RVT). 

Root Version structure: (variable number of entries) 

o VersionNumber |4 bytes] : Version number of the root directory, 
o FiIeNumber|4 bytes]: File number of the root directory, 
o VersionName [32 bytes]: Application version name. I.e. "SP 1". 
o Metadata [32 bytes] : See eStream FS Directory for format of the meta- 
data. 

3. Size Offset File Table (SOFT) section 

The SOFT table contains information to locate specific files in the CAF section. 
The entries are ordered according to the file number starting from 0 to Number- 
Files- 1. 

o NumberFiles (4 bytes] : Number of entries in this section. 

SOFT entry structure: (variable number of entries) 

o Offset |8 bytes] : Byte offset into CAF of the start of this file, 
o Size (8 bytes]: Byte size of this file. The file is located from address Off- 
set to Offset+Size. 

4. Concatenation Application File (CAF) section 

CAF is a concatenation of all file or directory data into a single data structure. 
Each piece of data can be a regular file, an AppInstallBlock, an eStream FS 
directory file, or an icon file. 
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a. Regular Files 

o FileData [X bytes]: Content of a regular file 

b. ApplnstallBlock (See ApplnstallBlock document for detail format) 

A simplified description of the ApplnstallBlock is listed here. For exact detail 
of the individual fields in the ApplnstallBlock, please see ApplnstallBlock 
Low-Level Design document. 

o Header section [X bytes]: Header for ApplnstallBlock containing infor- 
mation to identify this ApplnstallBlock. 

o Files section [X bytes]: Section containing file to be copied or spoofed. 

o AddVariable section [X bytes]: Section containing system variables to 
be added. 

o RemoveVariable section (X bytes]: Section containing system variables 
to be removed. 

o Prefetch section [X bytes]: Section containing pointers to files to be pre- 
fetched to the client. 

o Profile section [X bytes]: Section containing profile data, (not used in 
eStream 1.0) 

o Comment section |X bytes]: Section containing comments about Appln- 
stallBlock. 

o Code section [X bytes]: Section containing application-specific code 
needed to prepare local machine for streaming this application 

o LicenseAgreement section [X bytes]: Section containing licensing 
agreement message. 

c. EStream Directory 

An eStream Directory contains information about the subdirectories and files 
located within this directory. The information includes file number, names, 
and metadata associated with the files. 

o MagicNumber [4 bytes]: Magic number for eStream directory file, 
o StringTable |4 bytes]: Byte size offset to beginning of the string table, 
o StringTableLength [4 bytes]: Byte size length of the string table, 
o ParentFilelD [1 6+4 bytes]: AppID+FileNumber of the parent directory. 

AppID is set to 0 if the directory is the root, 
o SelfFilelD 116+4 bytes]: AppID+FileNumber of this directory, 
o NumFiles [4 bytes]: Number of files in the directory. 

Fixed length entry for each file in the directory: 

o FilelD [16+4 bytes]: AppBD+FileNumber of each file in this directory, 
o NameHash (4 bytes]: Hash value of the file name. Algorithm TBD. 
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o FHeNameOffset |4 bytes]: Offset where the file name is located, relative 
to the beginning of the string table. 

o FileNameLengtb |4 bytes]: Byte size length of the file name that is null- 
terminated. 

o Metadata (32 bytes]: The metadata consists of file byte size (8 bytes), 
file creation time (8 bytes), file modified time (8 bytes), attribute flags 
(4 bytes), eStream flags (4 bytes). The bits of the attribute flags have 
the following meaning: 

■ Bit 0: Read-only - Set if file is read-only 

■ Bit 1 : Hidden - Set if file is hidden from user 
The bits of the eStream flags have the following meaning: 

■ Bit 0: ForceUpgrade - Used only on root file. Set if client is 
forced to upgrade to this particular version if the current root ver- 
sion on the client is older. 

■ Bit 1 : RequireAccessToken - Set if file require access token be- 
fore client can read it. 

■ Bit 2: IsDirectory - Set if the file is a eStream Directory. 

d. Icon files 

o IconFileData [X bytes]: Content of an icon file. 
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Open Issues 



o Where is the metadata associated with the Root directory located? Cur- 
rently, root metadata is located in the root version table. All other files 
and directory metadata can be found in their parent directory. 
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Functionality 

The File System and Registry Filter Driver (FSRFD) is a part of the Builder module that 
monitors file system and registry updates initiated by the Builder process. This driver just 
intercepts such requests and records them and returns the recorded data to the Install 
Monitor (JNSTALLMON described in another LLD) program when requested by the lat- 
ter. All the intelligence, such as any decision-making logic, resides in the INSTALL- 
MON. 

For registry updates such as add or modify, the FSRFD needs to record the value added 
or modified. For registry deletes only the value name needs to be recorded. For file up- 
dates, there is no need to record the file contents added or modified, since the Builder 
would be interested only in the final contents of a file. 

Note: 

1 . This does not cover those rare cases where an existing file is updated by an 
application install, and the eStream client would need to make the same up- 
dates. This kind of functionality is difficult to implement and will not be con- 
sidered for 1.0. 

2. The FSRFD will be used to monitor only one install at a time to simplify the 
design of the FSRFD. That means you cannot invoke multiple instances of 
the Builder at a time to monitor multiple installations. All Builder invoca- 
tions on a machine have to be strictly sequential. 

3. This design is based on the driver model for WinNT and Win2K. The Win98 
driver is not covered here (yet). 

Data type definitions 

The following struct is used to communicate information related to activating the 
FSRFD. Specifically, the process-id of the INSTALLMON and the 2 drives whose ac- 
cesses need to be monitored are passed. 

struct MonitorActivate_t { 

ULONG processld; // PID of INSTALLMON 
UCHAR sysDrive; // System drive letter 
UCHAR destDrive; // Dest drive letter 

}; 
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The following struct is used to return monitored data back to the INSTALLMON. Note 
that this is a variable size struct where the last field keyName is an array of one wide- 
char, but in reality is an array of length whose value is the sum of 3 length fields in the 
struct (nameLength, valueNameLength^nd^ataLength). 

struct IMON_ENTRY { 

UCHAR regOrFile; // 'R' for registry, «F» for files 

// and *E' for end of data 
UCHAR updateType; // 4 A' for add, 'D' for delete, 

// «U' for update 
UCHAR valueType; // for registry only: value type 
// REG_SZ, REG_DWORD, REG_B I NARY , 

// REGJDWORD_LITTLE_ENDIAN, REG_DWORD_B I G_END I AN , 
// REG_EXPAND_SZ, REG_LINK, REG_MULTI_SZ, REG_NONE, 
// REG_QWORD, REG_QWORD_LITTLE_ENDIAN , 
// REG_RESOURCE_LI ST 

ULONG nameLength; // length of name (file or 

// registry key) in wchars 
ULONG valueNameLength; // length of value name (if 

// it exists) in wchars 
ULONG dataLength; // length of data in bytes 

WCHAR keyName [1]; // keyName followed by 

// valueName followed by 
// data: note none of these are 
// null terminated & are wide 
/ / chars 

}; 

Note about the updateType field: 'A' is used for file creation and 'U' is used for any up- 
dates to the file. So if a *U' is seen without an 'A' for a file that means the file was modi- 
fied but not created in this session. 

The following struct is used as the device extension in the FSRFD devices. Note that this 
extension is used for all device objects: the device that is created in the DriverEntry func- 
tion to represent an "INSTALLMON" device for the INSTALLMON to access our driver 
as well as the devices that are created to create a filter layer above existing drives. 

enum DEVI CE_TYPE { 

INSTALLMON I NTER FACE , 
STANDARD 

}; 
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struct DeviceExtension_t { 

PDEVICE__OBJECT deviceObject ; // device 

// for lower layer 
DEVICEJTYPE type; // see design 
KMUTEX pDeviceMutex; 
ProcessIdList 

// pointer to list of process-ids we are 

// interested in monitoring 

EntryList 

// This is the list that stores all the 
// info captured by the FSRFD until each 
// entry is queried by INSTALLMON 



There is an array for Fastlo that stores all the Fastlo function pointers which is required in 
a file system filter driver such as this. Note that we need to provide entry points for all (or 
most?) of the Fastlo routines in the dispatch table, since we need to pass the request down 
to the lower layer driver even if we are not interested in intercepting the request. For only 
some of the requests (e.g. all the FASTIO_* WRITE* requests), we would be recording 
the file access and creating an entry to be returned to INSTALLMON. The Component 
Design section describes in more detail the FastlO routines that are implemented by this 
driver. 



Interface definitions 
INSTALLMON interfaces 

Since the FSRFD is a driver, it cannot provide directly callable APIs. Instead the IN- 
STALLMON communicates with the FSRFD using the DeviceloControl Win32 API. It 
uses Ioctl codes MONACTIVATE, MONDE ACTIVATE and MONGETENTRY. 
The DeviceloControl API looks as follows: 



BOOL DeviceloControl ( 
HANDLE hDevice, 
DWORD dwIoControlCode, 
LPVOID lpInBuffer, 
DWORD nlnBuf ferSize, 
LPVOID lpOutBuffer, 
DWORD nOutBuf ferSize, 
LPDWORD lpBytesReturned, 
LPOVERL APP ED IpOver lapped 



// handle to device 
// operation control code 
// input data buffer 
// size of input data buffer 
// output data buffer 
// size of output data buffer 
// byte count 
// overlapped information 



The semantics of each of the MON_* codes is defined below. Note that this is described 
from the caller's (i.e. INSTALLMON) point of view. 
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Ioctl 1 - MONACTIVATE 

Input : 
hDevice : 

is the handle to the FSRFD device created. 

dwIoControlCode : 

The value MON__ACTI VATE 

IpInBuf f er : 

Address of MonitorActivate_t struct. This 
contains the process id of INSTALLMON. 

nlnBuf f erSize : 

Sizeof (MonitorActivate_t) 

Output : 

IpOutBuf f er : 

Should be a ptr to a ULONG (at least) . 

nOutBuf f erSize : 

Size of above buffer 

lpBytesReturned : 

Should be a ptr to a DWORD where byte count 
of data returned in IpOutBuf fer is returned. 

Comments: 

MON_ACT I VATE is sent to the FSRFD when the IN- 
STALLMON wants to start monitoring an installa- 
tion. MON_ACTIVATE can be sent only when the 
FSRFD is not already active - either after the 
driver is loaded the first time or after the 
last MON_DEACTIVATE request. 

Errors : 

Q STATUS_ALREADY_ACTI VE : FSRFD was already 
activated. The processld of the old acti- 
vation is returned in the ULONG pointed by 
IpOutBuf fer. 

□ STATUS_INVALID_ARG: One of the arguments 
passed is not valid (either invalid Moni- 
torActivate__t ptr or processld) . 
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□ STATUS_INVALIDJ3RIVE: Either the sysDrive 
or the destDrive (or both) is invalid. 



Ioctl 2 - MON DEACTIVATE 

Input : 

hDevice : 

is the handle to the FSRFD device created. 

dwIoControlCode : 

The value MON_DEACTIVATE 

lpInBuf fer : 

NULL, or pointer to a ULONG where the 
ULONG has a non-zero value indicating a 
"forced" deactivation. A "forced" deacti- 
vation is done when the FSRFD still has 
entries that are not going to be re- 
trieved. 

nlnBuf f erSize : 

0 or size of ULONG (depending on the 
above) . 

Output : 

No need for OUT arguments. 

Comments : 

MON_DEACTIVATE is sent to the FSRFD when the 
INSTALLMON wants to stop monitoring an instal- 
lation. MONJDEACTIVATE can be sent only when 
the FSRFD is already active i.e. after the last 
MON__ACT I VATE request. 

Errors : 

□ STATUS_NOT_ACTIVE: FSRFD was already deac- 
tivated. No special action is needed to be 
taken for this error condition. The caller 
can simply send a new MON_ACTIVATE . 

□ STATUS_PENDING_DATA: The FSRFD has some 
entries that were not read using 
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MON_GET_ENTRY . This error is only returned 
in case of non-forced deactivation. 

Ioctl 3 - MONGETJENTRY 

Input : 
hDevice : 

is the handle to the FSRFD device created. 

dwIoControlCode : 

The value MONJ3ETJENTRY 

lpInBuf f er : 
NULL. 

nlnBuf f erSize : 
0 

Output : 

lpOutBuffer: 

Should be a ptr to a IMON_ENTRY struct. 

nOutBuff erSize : 

Size of above buffer 

IpBytesRetumed : 

Should be a ptr to a DWORD where byte count 
of data returned in lpOutBuffer is returned. 

Comments: 

MON_GET_ENTRY is sent to get the next M entry" 
from the FSRFD. An "entry" is a record of a 
registry or file update intercepted by the 
FSRFD. The details are returned in the 
IMON_ENTRY struct passed in the lpOutBuffer ar- 
gument. Note that the FSRFD uses an event ob- 
ject (created using the IoCreateNotif icat ion- 
Event API) to signal the INSTALLMON that an 
"entry" is available to be read. INSTALLMON 
waits on this event object before retrieving 
the entry using MON_GET_ENTRY . 

Errors : 
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□ STATUS_NOT_ACTIVE : FSRFD was not acti- 
vated . 

□ STATUS_INVALID_ARG: One of the pointer ar- 
guments passed is not valid. 

□ STATUS_INSUFF_BUFFER : The buffer size in- 
dicated by nOutBuf ferSize is insufficient 
for the current "entry" data. 

IoctI4 - MONGETERROR 

We need this to indicate occurrence of an error when- 
ever this occurs in any of the dispatch functions. We 
can either implement this control code or just return 
an error code for any M0N_GETJ3NTRY call that reflects 
that an error occurred. 

Event Object interface 

As mentioned above, an event object (lets call it IMON event object) will be used to sig- 
nal the INSTALLMON that a new entry is available. This event object will be created in 
the FSRFD in the processing of M ON_A CTIV ATE i octl , as: 

ext->pEvent = IoCreateNotif icationEvent ( 
L " \ \BaseNamedOb j ec t s \ \ INSTALLMONEVENT " , 
ext->eventHandle); 

Whenever the FSRFD has a new entry, it signals using the above event object as follows: 
KeSetEvent (ext - >pEvent , 0 , FALSE) ; 

Whenever INSTALLMON gets the next entry using M0NJ3ETJENTRY ioctl, the 
FSRFD resets the event (if the list of entries is going to be empty after this entry) as fol- 
lows: 

KeClearEvent (ext->pEvent) ; 
Whenever a MONJDEACTTVATE is processed, the FSRFD will close the event as: 
ZwClose (ext->eventHandle) ; 

Kernel or Low Level Driver Interfaces 

Every driver needs a DriverEnt ry routine that is called when the driver is first loaded. 
This routine for FSRFD is described in the Component Design section. 
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The FSRFD inserts "hook routines" that intercept relevant registry and file system calls. 
The Component Design section describes which hook routines are inserted and what they 
do. 

Component Design 

Global variables 
plmonDevice 

This global variable points to the device object created in the DriverEntry function for the 
"WDeviceWinstallmon" device. 

DriverEntry 

NTSTATUS DriverEntry (IN DriverObject , IN RegistryPath) 

IoCreateDevice for u \\Device\\installmon" ; 
plmonDevice = pDeviceObj ect returned above; 
ext = plmonDevice- >DeviceExtension ; 
ext->type = INSTALLMONINTERFACE ; 
IoCreateSymbolicLink with 

M \\DosDevices\\installmon" ; 
for all IRP_MJ_* values upto 

I R P_M J_MAX I MUM_FUNCT I ON { 
DriverObject->MajorFunction [IRP_MJ_*] = 
ImonDispatch 

} 

Setup the unload driver function 
DriverObject->FastIoDispatch = address of 
our fast io dispatch table ; 
Note that we are interested only in 
FAST_IO_*WRITE* routines for getting our 
entries, however we need to implement all 
of them to call lower layered drivers. All 
our Fastlo funcs are called ImonFastlo*; 
Create the necessary mutexes; 

Use PsSetCreateProcessNotifyRoutine to set a 
process create callback routine 
ImonProcessCallback; 

} 

ImonProcessCallback 

{ 

// similar to ImonProcessCallback 

// This function is called every time a 
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// process on the system is created or 

// deleted. We need to figure out (by 

// checking the parent id in our list) 

// if we need to add or remove this process 

/ / from our list 

lock the mutex for ProcessIdList 
if not activated just return; 
if (this is process create) { 

Look up the parent process id in the 

ProcessIdList 

If present, add this process id to the 
ProcessIdList 

} 

else { 

if this process id is present then 
remove the process id 

} 

release the mutex 

} 

ImonDlspatch 

{ 

// similar to FilemonDispatch 

// Instead of registering a different 

// function for each IRP_MJ_* this function 

// is called for all of them and this one 

// dispatches the right on based on the 

/ / control code 

This gets called for all IRP_MJ_* ; 
if the device extension type tells us 
I NSTALLMON I NTER FACE 
call I monDe vi ce Func 

else 

call ImonHookFunc 

} 

ImonDeviceFunc 

{ 

// similar to FilemonDeviceRoutine 
// This function is called whenever an Ioctl 
// comes from the Installmon process that is 
// meant to be a command for this FSRFD. 
This is a request from the INSTALLMON using 
the INSTALLMON device. We would mainly be 
processing IRP_MJ_DEVICE__CONTROL, although 
ImonFastloDeviceControl should have been 
called. So if we come here just call 
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ImonFastloDeviceControl 
Cal 1 IoCompl et eRequest ? 



ImonHookFunc 

{ 

// similar to FilemonHookRoutine 

// This is the hook function that is called 

// for all the I/O requests that is made by 

// the I/O manager that need to go through 

// this driver (i.e. for those requests 

// that we are filtering) . 

We are interested in recording: 

I RP_M J_CREATE where the Irp-> 

Parameters . Create . Options indicates 
a new file create (as opposed to an 
existing file open) 

IRP_MJ_WRITE 
Note that we have to get the current 
process- id using PsGetCurrentProcessId and 
look it up in the ProcessIdList (note: you 
have to exclude the first process- id since 
that belongs to the Installmon and not the 
setup program) and only if that search is 
successful, record the entry 
Note that we need to get the filename from 

the FileObject using code similar to 

F i 1 emonGe t Fu 1 1 pa t h 
Also we should be recording the entry when 
the request is successfully completed. So do 
this in the completion routine in a manner 
similar to filemon. 
To record: 

create a relevant IMON_ENTRY record; 

Call ImonAddEntry with this record; 
Pass all Irps to lower layer driver using 

IoCallDriver and getting the lower layer 

device object from this device's 

ext->deviceObj ect 

J 

ImonFastloDeviceControl 

{ 

// similar to both 

// FilemonFastloDeviceControl and 
/ / I monD i spa t ch I oc 1 1 

// This function gets called for all the 
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// IOCTLs. If it is from Installmon, we need 
//to process the MON_* commands or else 
// just pass on the command to the lower 
// layer driver. 

Get the current device's extension 
if type indicates I NSTALLMON INTERFACE { 
switch (IoControlCode) { 
case MON__ACTIVATE : 
call ImonActivate; 
break; 
case MON_DfiACTIVATE: 
Call ImonDeactivate; 
break; 
case MON_GETENTRY : 
Call ImonGetEntry; 
break; 

} 

} 

else { 

pass it down using deviceObject and the 
fastio hook for Ioctl 

} 

} 



ImonAddEntry 

{ 

// similar to ImonEnqueueRegEntry and 
// createRegEntry etc. 

// This function adds a IMON_ENTRY node 

// to our list: this list is eventually 

// returned to InstallMon 

create an entry rec from nonPagedPool 

Grab a mutex to modify EntryList 

Add the entry rec to EntryList 

release the mutex 

KeSetEvent for IMON event obiect 

} 

ImonActivate 

{ 

// similar to ImonActivate and various 

// Filemon funcs called by 

/ / FilemonFastloDeviceControl 

// This is called by our own func that 

// handles IOCTLs when a MON_ACTIVATE is 

// sent by InstallMon 

Look at the ProcessIdList to ensure that we 
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are not already active (1 st process id) . 
If we are, return with an error with that 
process id 
Grab a mutex 

Add a node to ProcessIdList with the 

processld; 

Release the mutex; 

Create the IMON event object; 

HookDrive (sysDrive) ; 

HookDrive (destDrive) ; 

HookRegistry () ; 

} 

ImonDeactivate 

{ 

// Same as above except for MONJDE ACTIVATE 
If we are already deactivated 

return with an error; 
If EntryList is not empty and this is not 

a forced deactivation then 

return with error; 
Clear and Delete the IMON event object ; 
Free the ProcessIdList ; 
Free the EntryList; 
UnhookRegistry ( } ; 

} 

ImonGetEntry 

{ 

// When the InstallMon sends a MON__GET_ENTRY 

// this function is called. 

Grab the mutex to access EntryList 

get next entry and remove from the 

list; 

if no entry then { 

if (deactivated and first process in 
the processIdList is dead) { 
Make a new entry of "end of data" 
type ; 

} 

else { 

there is some problem (should not 
happen) 

} 

} 

copy the entry into the OUT buffer 
release the mutex ; 
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} 

HookDrive, UnhookDrive 

{ 

// very similar to Filemon's HookDrive 
// Hence not described here 
// Similarly UnhookDrive 

// When a MON_A CT I VATE comes, we need to 
// make sure that all the requests to the 
// system drive or dest drive are filtered 
// through us. e.g. If n C:" is the system 
// drive, HookDrive will register this 
// as the filter driver for all 10 for W C:" 
// Similary for UnhookDrive. 

} 

HookRegistry. UnhookRegistry 

{ 

//similar to Installnion' s (Un) HookRegistry 
//We need to make sure that all Registry 
// functions are replaced, with our funcs 
//so that these funcs get called whenever 
// a process is trying to access the 
// registry. Our hook funcs will in turn 
// call the real funcs after queueing an 
// entry . 

} 

ImonFastlo* routines 

{ 

// These are very similar to FilemonFastlo 

// routines except that we need to intercept 

// only FAST_IO_WRITE, 

// FAST_IO_MDL_WRITE_COMPLETE, 

// FAST_IO_WRITE_COMPRESSED, 

// FAST_I0_WRITE_C0MPLETE_COMPRESSED 

Call the lower layer driver; 

if the request was successfully completed { 
Record the details into our entry rec 
and enqueue it similar to how ImonHookFunc 
does it; 

} 

} 

Interesting issues to deal with: 
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□ Make sure that we use non-paged memory as required, e.g. all the nodes of the 
processIdList and entryList will be from non-paged pool. According to Bob, we 
do not need to always allocate non-paged memory - for Registry related nodes 
(i.e. when it is *R' type 1MON ENTRY node) we can allocate paged memory. 
However this needs to be checked - since there will be pointers between these 
nodes, we need to make sure that this will work properly. 

□ We have to make sure that a 2-phase install works with this: some setups ask you 
to reboot the machine and after the reboot the setup continues. For the FSRFD we 
need to make sure that after the reboot the FSRFD is already loaded before the 2 nd 
phase of the install starts. In the FSRFD we may need to make sure that when the 
"Runonce" registry key is updated (the installer is trying to do a 2-phase install 
and setting the 2 nd phase exe as the value of "Runonce") we capure that info and 
accordingly co-ordinate with the Installmon to do the right thing. To ensure that 
this driver is started at the right time on start-up (since the Builder machine is go- 
ing to be an in-house dedicated machine, it is okay), we need to add to the System 
registry (under 

HKEY__LOCAL_MACHINE\SYSTEM\Current Control Set \ Services) 
appropriate values for Start, Group and Tag (and possibly others) value 
names. Actually this is going to be implemented in InstallMon as a user initiated 
event in which case the user informs the Builder UI (which in turn informs the In- 
stallMon) that a reboot is imminent. In that case the InstallMon can try to find all 
possible ways in which the setup.exe is achieving this: 

o The "RunOnce" key or one of its other incarnations (e.g. RunOnceService, 

RunOnceEx etc) has been modified. We need to figure out exactly which 

one of the "*RunOnce*" can be modified, 
o The setup.exe has actually added itself (or another exe) to the startup 

folder. If that is the case, we can do the same trick here: replace that entry 

with an entry pointing to the BuilderUI with the original setup.exe value 

as an argument to the BuilderUI. 

□ An issue that hasn't been resolved is any user interaction (and. user input) that has 
taken place during installation that is being monitored. For example, an installa- 
tion may ask for a port number that it may store in a registry key. The solution 
suggested is as follows: This has to be a manual process. The Builder user should 
record all the manual interaction and manual data input that has taken place. He 
should recreate the same interaction in the custom DLL that the appInstallBlock 
provides for that app. This custom DLL at eStream app subscribe time can do the 
same thing that was observed during original application install. Alternatively we 
can request the ISV's co-operation in doing this. (May be this bullet should be 
transferred to a different doc such as the InstallMon LLD.). 

□ There is another unresolved issue about handling h/w or s/w dependent things that 
the installer does: we will have to handle this case by case basis and any knowl- 
edge we gain as a result of this, we should consolidate in the Builder components, 
e.g. we may notice that some installations may depend on IE4 or IE5 being there. 
Of course, one of the pre-requisites of the Builder is that it will be run on a pris- 
tine machines, so that we capture a maximal installation when it is taking place. 
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Testing design 
Unit testing plans 

This will be unit tested using the INSTALLMON program (or its early prototype). The 
INSTALLMON program sends all the required Ioctls like MONACTTVATE 
MONDEACTIVATE and MONGETENTRY. The INSTALLMON output will be used 
to check the correctness of the FSRFD. This means the INSTALLMON itself should be 
assumed to be correct. 

In addition to the above, we actually need to write one or more test programs that exer- 
cise the FSRFD. These test programs will be run as if they were App Installers i.e. as 
child processes of the INSTALLMON. The test programs should be written to exercise: 

□ All file systems (FAT, FAT32, NTFS, HPFS, compressed drives and others), 
since the Fastlo routines need to be tested. 

□ For each of the file systems: 

o File create (with and without the old file being there) 
o File update (existing file appended as well as modified in the middle) 
o File touch (e.g. get the version of a DLL file) - this is not yet covered by 
this design 
o Registry updates: 
o create a key 
o delete a subkey 
o delete a named value from a key 

o RegReplaceKey (we need to investigate if this is broken down into smaller 
reg calls). 

o RegRestoreKey (same comment as above applies) 

o RegSetKeySecurity (we failed to address this in the design) 

o RegSetValueEx 

o RegLoadKey and RegUnLoadKey 

Stress testing plans 

The FSRFD will be stress-tested using the above testing strategy by varying the rate at 
which files/registry are updated and under a variety of conditions (memory/disk, other 
processes). 

Coverage testing plans 

The unit testing above also covers Coverage testing. 
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Cross-component testing plans 

This will be tested with the Installnion program which is enough for cross-component 
testing. 

Upgrading/Supportability/Deployment design 

Deployment: This will be used in-house, so no deployment considerations. 

Open Issues 

These issues are not necessarily FSRFD related, but are listed here just to remind our- 
selves. 

□ Do we need to worry about environment variables? It looks like most installations 
(and their apps) won't be looking at environment variables. 

□ What about the .Ink files (shortcuts). It looks like the client guys will have to 
change all the .Ink files based on the actual client's settings. This issue has been 
addressed either in the InstallMon or Packager. Pis see those documents. (The 
IShellLink interface has been suggested). 

□ Also what about when device drivers are installed? There is no impact on the 
builder but the client will need to reboot and hopefully the existing applnstall- 
Block and the custom initialization dll should be able to take care of it. 
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eStream Installmon Low Level Design 

Sanjay M Pujare 
<Date> 

Functionality 

The Installmon is a part of the Builder module that talks to the FSRFD to monitor file 
system and registry updates initiated by the Builder process. The FSRFD driver just in- 
tercepts such requests and records them and returns the recorded data to Installmon when 
requested by the latter. All the intelligence, such as any decision-making logic, resides in 
the Installmon. 

Note: 

1 . This does not cover those rare cases where an existing file is updated by an 
application install, and the eStream client would need to make the same up- 
dates. This kind of functionality is difficult to implement and will not be con- 
sidered for 1.0. 

2. Somewhere in the docs (may be the user docs?) it should be mentioned that 
the Builder's (or rather Installmon 5 s) job could be made easier by the user 
by following these guidelines: 

- Make sure that values in all the registry keys are as distinct as possi- 
ble. We may need to create a special Win2K or WinNT installation 
where each key (or ValueName within a key) will be created with a 
distinct or unique value as far as possible. E.g. If the default Windows 
installation creates 2 keys FOO and BAR and stores the same value 
"C:\Windows\System32" in both of them, we wouldn't know which 
one of those keys is used when a file is copied to 
"C:\Windows\System32". To solve this problem, all the effort should 
be made to ensure that FOO and BAR have distinct values. 

- When the Builder operator is installing an app under the Installmon, 
she should also make sure that the install script is given a distinct or 
unique value for each of the user inputs that may be used to set a reg- 
istry key or an environment variable. This is especially necessary 
when the inputs seem to be totally unrelated. For example, vendor 
name and installation directory name. If both are entered as "Micro- 
soft" then that could cause confusion to the Installmon. 

3. There are 2 ways in which registry changes and file system changes can be 
captured: 

□ using a kernel mode driver such as the FSRFD, Or 

□ using a diffing mechanism for both the registry as well as the file system. 
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In this design, we use both the approaches, and record any differences due to 
these two approaches. We give the user a chance to manually edit the regis- 
try/file changes. 

Data type definitions 
Interface definitions 

All of the interfaces used to communicate with the FSRFD are described in the FSRFD 
LLD document. This LLD document will cover only interfaces with the other modules. 

Interfaces with the Builder UI 

The public methods of the InstallMon class described below are interfaces to the Builder 
UI. Specifically these are: 

void InstallMon: :startCapture (PUNICODE_STRING setup_exe, 
PUNICODEJSTRING dest, bool upg, FileTable_t *fTbl); 

This function is called when the user chooses to start monitoring an install. This function 
is called after the user has entered all the necessary data such as the setup.exe path etc. 

void InstallMon: : stopCapture ( ) 

This is called in rare cases when the user wants to terminate a running install in abnormal 
cases. 

bool InstallMon: : checkSetupStatus ( ) 

This is used by the UI to check the status of the install: whether the file list is ready to be 
processed and the registry list is ready to be processed etc. 

void InstallMon: :getRegistryList ( ) 

This function is used to finally get the list of registry changes that occurred after the in- 
stall has taken place. This function allows the user to edit the list to manually 
add/delete/modify entries to be able to override. 

void InstallMon: :getFilesList () 

This function is used to finally get the list of file system changes that occurred after the 
install has taken place. This function allows the user to edit the list to manually 
add/delete/modify entries to be able to override. 

void InstallMon: :machineToBeRebooted () 

When the app install program is asking the user to reboot the machine (in the middle of 
the install) to continue installation, the user has to inform the Builder UI that a reboot is 
imminent. The InstallMon does the necessary bookkeeping to make sure that monitoring 
continues after the reboot. 

void InstallMon: : startCaptureAf terReboot (setup_exe , dest, 
upg) 
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When (he installation continues after the reboot, the Builder Ul is automatically invoked, 
and it calls this function to inform the InstallMon to continue monitoring. 

Access DB interface 

We will be using an Access DB (or alternately the MSI databases as suggested by 
Bhaven) to store intermediate results of the Installmon process. There will be 2 tables 
used: Registry and Files. 

Registry table 

Fullpath : string; // this is the full path of the key 

ValueName : string; // this is the value name: for 

// (default) use NULL 
UpdateType: char; // Add/update or delete, also 

// X R' for removed 
// combined (Fullpath, ValueName) should be unique i.e. 
// primary key 



Files table 

Fullpath : string; 

UpdateType : char; // add, update 

Kind: char; // 'Copied, ^S'poofed, ^E' stream 

Fileld: Number; 

// Combined (Fullpath, UpdateType, Fileld) should be unique 

RegistryDiff table 

Fullpath : string; 
ValueName : string ; 
ValueType: char; 

Value: something that can store all REG_* types 
status: char; // Original, *C (add/change),, *D' 
// combined (Fullpath, ValueName) should be unique 

FilesDiff table 

Fullpath: string; 

Type: char; // file % F' or dir % D' etc 

Status: char; // *0'riginal, ...'C, y A' , *D' etc 

// (Fullpath 

Component Design 

struct FileTable_t { 
PUNICODE_STRING name; 
Char type; // spoofed, copied or eFS 
Int fileld; 

}; 



class InstallMon { 
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PUNICODE_STRING setup, destDrive; 
bool interruptThread; 

Event Object signalMoni tor , signal Setup; 

bool rebootReq = false; 

bool afterReboot = false; 

bool completed ; 

int exit Code; 

bool upgrade ; 

some Type envVars ,- 

// TODO: combine the above 2 into an enum 
int upgradeFileldBegin = -1; 
int currFileld; 

static threadSetup ( InstallMon iMon) { 

Remove all the environment variables except 
the ones in iMon->envVars; 
Start the Setup program; 
And wait for it to finish; 

iMon->exitCode = exit code from the process; 
when finished signal the signalSetup event; 

} 

static threadMonitor ( InstallMon *iMon) { 

// this is the func that runs as a separate 
// thread, until we are interrupted or we 
// notice that the setup process has exited, 
get the current 

process id, system drive (using 

GetSystemWindowsDirectory) , destDrive and send 
the MON_ACT I VATE message> 
Start threadSetup thread with setup_exe; 
processEnded = false; 
while ( ! interruptThread) { 
if (rebootReq) { 

only poll for Installmonevent ; 
if (signal not set) { 

break from the while loop; 

} 

} 

else { 

WaitForMultipleObj ects -> signalMonitor , 
Installmonevent and signalSetup; 

} 

switch (signal) { 
case Installmon event: 

get the MonitorEntry__t record; 

switch (regOrFile) { 

case X R' : 

switch (updateType) { 
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case 'A' : 
case *U' : 

add or update (KeyName, ValueName, 
V N') to 
registry table; 
break; 
case *D' : 

add (KeyName, ValueName, X D' ) to 
registry table; 

} 

break ; 
case % F' : 

switch (updateType) { 
case *A' : 

// file creation 

add (fullpath, V A' , *?', 

iMon->currFileId++) to files 
table ; 
break; 
case l U / : 

add (fullpath, *U') to files 

table; 
break; 

} 

break; 
case *E' : 

no more data to add; 
if ( iprocessEnded) { 

// something wrong! ! ! ! 

} 

break out of the while loop 
break; 

} 

break; 
case setupevent : 

processEnded = true; 

Send MON_DEACTIVATE to the FSRFD; 

break; 

case signalMonitor event: 
if (rebootReq) { 

kill the threadSetup thread 
clear the signalMonitor event; 

} 

else { 

// TBD? 

} 

break; 
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} // switch 

} // while 

if (processEnded) { 

// normal setup process completion 

// registry and files tables should have 

// all the captured data from the FSRFD 

iMon- >diff Capture ( ) ; 

iMon- >completed = true; 

} 

} // threadMonitor 

void commonCapture (setup_exe, dest) { 
setup = setup_exe; 
destDrive = dest; 

Start the threadFunc thread, and pass 'this' to 
it; 

> 

void diff Capture () { 

// Bhaven suggested that we could instead do an export 

// from regedit and do a text diff for registry diffs. 

// similarly: It seems the regular Unix diff tool also 

// compares directories. I don't know if it is 

// recursive. If it is, we might be able to use it. 

// Further investigation is recommended for doing 

// files diff. 

Query the OTI \BUI LDERSTART key and get the value 

in builderStartTime and delete the 

OTI \BUILDERSTART key; 

// process the registry 

// step 1: query RegistryDiff table 

for each row in RegistryDiff table { 

query the corresponding key+valueName in the 

Windows registry 

if (exists) { 

if (no change to value) { 

update the row status to *U' for unchanged 

else . { 

update the row status to y C (changed) 

} 

} 

else { 

update the row status to 'D' (deleted) 

} 

} 

// step 2: enumerate the whole Windows registry 
for each enumerated registry key+value { 
query the RegistryDiff table; 
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if (a row exists) { 
if (status is 'U' ) { 
delete the row 

} 

else { 

status should be *C and values should be 
different bet table and actual registry 
or else display fatal error (?) 

} 

} 

else { 

// no previous value 

add a row to RegistryDiff with (fullpath, 

valueName, ValueType, Value, X A' ) to mark 
it as an added registry key 

} 

} 

// now repopulate the FilesDiff table 
// step 1: query FilesDiff table 
for each row in FilesDiff table { 

query the corresponding Fullpath in the actual 
file system; 
if (exists) { 

if (no change (i.e. timestamp before 
builderStartTime) ) { 

update the row status to *U' for unchanged 

} 

else { 

update the row status to X C (changed) 

} 

} 

else { 

update the row status to *D' (deleted) 

} 

} 

// step 2: traverse the whole file system 
for each file/dir in (systemDrive, destDrive} { 
query the FilesDiff table; 
if (a row exists) { 
if (status is 'U') { 
delete the row 

} 

else { 

status should be % C and the file/dir 
timestamp should be after builderStartTime 
or else display fatal error)?) 

} 



Omnishift Technologies, Inc. 



7 



Company Confidential 



eStream <COMPONENT> Low Level Design 



} 

else { 

// no previous value 

add a row to FilesDiff with (fullpath, 

'F' or X D', 'A') to mark it as an added 
f ile/dir ; 

} 

} 

} 

void recursiveFileGetter (curDir) { 

add a row to FilesDiff as (curDir, *D' , *0'); 
for (each element of curDir) { 
if (element is a dir) { 

recursiveFileGetter (element) ; 

} 

else { // has to be a file(?) 

add a row to FilesDiff as (element, *F' , 
^O'); 

} 

} 

} 

public : 

InstallMon(?) ; 
void setEnvVars() { 

allow the user to set environment variables 
in an interactive way; 
. Get the values in envVars; 

} 

void startCapture (PUNICODE_STRING setup_exe, 

PUNICODE_STRING dest, bool upg, FileTable_t *fTbl) { 

clear the Access DB registry, files, 
registryDiff and FilesDiff tables; 

if (upg) { 

populate the files table with data from the 

fTbl array; 
upgradeFileldBegin = last file id + 1; 
currFileld = upgradeFileldBegin; 

} 

else { 

currFileld = 0 or 1 (depends on where to start) ; 

Query the whole registry and 

for (each key and valueName pair) { 

add (key, valueName, valueType, value, y 0') to 
registryDiff; 

} 

Get the current time and store it in some format 
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in a registry key OTI \BUI LDERSTART ; 

for (curDrive in {systemDrive, dest}) do 

Root = root of curDrive (e.g. W C:\\"); 
recursiveFileGetter (Root) ; 

} 

interruptThread = false; 
afterReboot = false; 

commonCapture (setup_exe, dest, upg) ; 

} 

void stopCapture () { 

// abnormal capture termination 
// TBD 

// also there might be normal cases where this 
// func is called when the setup process exit 
// is not detected for some reason (or doesn't 
// happen) ! May be we don't have to worry about 
// these things. 

} 

bool checkSetupStatus () { 
if (! completed) { 

display error and return false; 

} 

if (exitCode is not okay) { 

display error and return false; 

} 

} 

void getRegistryList () { 

checkSetupStatus () and conditionally return; 
Using appropriate SQL, show (fullpath, ValueName) 
tuples that exist in Registry but not in 
RegistryDiff ; 
Tell the user that these were captured by FSRFD 
(installmon) driver but not by the diff process; 
If user chooses to remove any tuples from the 
shown list, mark these with status as *R' in the 
Registry table; 

Using appropriate SQL, show (fullpath, ValueName) 
tuples that exist in RegistryDiff but not in 
Registry ; 

Tell the user that these were captured by diff 
but not by the FSRFD (installmon) driver; 
If user chooses to add any tuples from the 
shown list, add these tuples to the Registry 
table with status copied from RegistryDiff; 
Remove all the rows from the Registry table 
where status is A R ' ; 
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Now we have all the correct entries in Registry; 
Read each row of Registry and into an array to be 
passed to the caller (most probably the. caller of this 
func would have passed an array in which we should 
pass these rows) / 

} 

void getFilesList ( ) { 

checkSetupStatus ( ) and conditionally return; 
Using appropriate SQL, show (fullpath) 
tuples that exist in Files but not in 
FilesDif f ; 

Tell the user that these were captured by FSRFD 
(installmon) driver but not by the diff process; 
If user chooses to remove any tuples from the 
shown list, mark these with status as X R' in the 
Files table; 

Using appropriate SQL, show (fullpath) 

tuples that exist in FilesDif f but not in 
Files; 

Tell the user that these were captured by diff 
but not by the FSRFD (installmon) driver ; 
If user chooses to add any tuples from the 
shown list, add these tuples to the Files 
table with status copied from FilesDif f; 
Remove all the rows from the Files table 
where status is 'R'; 

Now we have all the correct entries in Files; 

Using appropriate SQL, select files (and not 
dirs) where the file has only *U' and not *A' : 
this means the setup modified an existing file 
and we cannot handle this; so if this happens 
show a fatal error; 

Read each row of Files and classify it into % C , 

*S' or *E' based on the following logic: 

If the file belongs to the app install directory 

(or app drive?) it is an 'E'; 

If the file is smaller than a threshold then 
it is y C or else it is *S'; 

if (upg) { 

we need to create new file-ids for directories 

that contain new files or directories 

int prevStart = upgradeFileldBegin; 

int prevEnd = currFileld; 

while ( (prevEnd - 1) > prevStart) { 
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using appropriate SQL, select rows from the files 
table where fileld >= prevStart and kind is not 
*C; 

for each such row { 

fullpath = fullpath in the row; (e.g. "C:\A\B") 
dir = parent directory of fullpath (e.g. 
"C:\A" ) 

select in files a row where fullpath = dir and 
fileld >= upgradeFileldBegin; 
if (doesn't exist) { 

add in Files a row with (dir, *X' , *?', 
currFileId++) ; 

// Here *X' stands for "new file id for a 

// directory created because one of the 

// children has a new file id 

} 

} 

prevStart = prevEnd ; 
prevEnd = currFileld; 

} 

} 

Also for each file, change the absolute path to 

an envVar relative or registry-value relative 

one: this is a non-trivial task. The way to do this is 

proposed below. However we need to refine this 

algorithm based on actual Builder runs on real apps : 

For a basic Win2K (or WinNT as the case may be) system 
create a list of registry keys (and env vars) whose 
values are normally used as destination paths for 
copying various kinds of files. To this list also add 
this app's dest dir as one of the values. Also add all 
the registry keys/values added as part of this app's 
install. Create an access table that looks like: 

Key: String; // registry or env or other name 

Type: char; // Registry, *E'nv, x D'est dir etc. 

Source: char; // ^S'ystem, x A'pp, *U'nknown? 

Value: string; 

Now all of the *E' files should be relative to dest 
dir (i.e. 'D' type above). Also any sub-directory 
strings should either be hard-coded or based on some 
registry key values where the registry key is of 
source *A' (app) above. 
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All the *C and X S' files should be relative to one of 
the registry keys of source X S' above and preferably 
type *R' (since environment variables are not used un- 
der Windows?) . 

If there is a sacred file set (i.e. files that are 
so "sacred" for eStream apps, that they were installed 
on the client when the eStream client was installed) , 
we need to remove those files from this list; 

Set these values in the table and read the whole 
table in an array (most probably the caller of 
this func would have passed an array in which we 
should pass these rows) ; 

} 

void machineToBeRebooted ( ) { 

//The setup is going to reboot the machine. 

//Note that there is a race condition possible here. 

// Suppose the setup.exe has displayed a dialog 

// asking the user to confirm a reboot. At this time 

// the setup.exe has still not written to "Runonce" . 

// Only after the user has confirmed (by pressing 

// Okay) will the program write to "runonce" and 

// auto reboot. In that case, this function will be 

// called at the wrong time. We may need to modify 

// the FSRFD to detect changes to "Runonce" . 

reboot Req = true ; 

signal the signalMonitor event; 

wait for the threadMonitor thread to end; 

Query either our Access database or the actual 

registry to see if Runonce key is set/updated. 

if (not) { 

we cannot handle this reboot situation; 

give fatal error; 

// may be we can look at the 

// Start\Programs\Starup folder and do the 

// same thing we did with Runonce key 

} 

OldRunonce = Old value of Runonce key; 
Set the new value of Runonce key as our Builder 
name (or whichever EXE has the installmon code) 
followed by OldRunonce. as the argument to that 
EXE and the destDrive as the next argument and 
upgrade as the next arg; 

Prompt the user to go ahead and say Okay to 
Setup's dialog warning that it is going to 
reboot ; 
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} 

void start CaptureAf terReboot (setup_exe, dest, upg) { 
afterReboot = true; 

commonCapture (setup exe, dest, upg); 

} 

}; 

Interesting issues to deal with: 

Testing design 
Unit testing plans 

The unit testing of installmon will be done in conjunction with the FSRFD unit testing. 
This has been described in the FSRFD-LLD document. In addition, we will be using the 
sysdiff tool to validate the results of the Installmon. 

Stress testing plans 

All of the 14 or so applications (that OTI is planning to convert to eStream) installs will 
be tested under the installmon. Any issues found will be used to fix and improve the in- 
stallmon functionality. 

Coverage testing plans 

The testing of the Builder/installmon on the 14 or so app installs should give us enough 
coverage. Considering that Builder/installmon will be used in-house for some time makes 
some of the testing issues less significant. 

Cross-component testing plans 

Will be tested as a component of the whole builder. 

Upgrading/Supportability/Deployment design 

Deployment: This will be used in-house, so no deployment considerations. 

Open Issues 

□ On one of the newgroups someone mentioned a key 
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\SharedDLLs) that 
we can look at for the shared DLLs that cannot be installed. Need to investigate. 

□ 2-phase install: some installs need a reboot after which they continue. The key to 
look at is Runonce (under the same path as SharedDLLs I think). 
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Functionality 

This component is a device driver that is started and stopped by the eStreamCIient.exe 
program. 

The registry-spoofing database is a list of registry entries that the registry spoofer 
redirects whenever client programs or the windows kernel requests spoofed registry keys. 
The reason that we do this is to present different views of the registry depending on when 
different users are logged into the system. Upon completion of User Login a startup 
program called eStreamControl is begun to initialize die following client components 
(not necessarily in this order). 

1 . Registry Spoofing 

2. File Spoofing 

3. Z File System Set up 

4. eStream Logon and user Verification. 

Registry spoofing is begun by a call into an eStream Device Driver from the 
eStreamControl program that will start the kernel level registry spoofer. When the user 
logs out the eStreamControl will make a call to the eStream Device Driver to shut down 
the registry spoofing. If another user logs on to the same computer who is not enabled as 
an eStream user then that users view of the registry is identical to what it would be if 
eStream were not installed on the client computer. 



Data type definitions 



The registry entries to be spoofed are provided on a per-user set. Every user who logs on 
to a computer with eStream installed will have their own set of spoofed registry entries. 
These registry entries will be stored under HKEYCURRENTUSER. This will enable 
simple back up of the keys when the user logs out. The HKEY_CURRENT_USER Keys 
are automatically backed up to HKEYUSERS when the user logs out and restored when 
the user Logs back in. 

Keys that need to be spoofed are always in one of the following places 

HKEY_CLASSES_ROOT (HKCR) 
HKEY LOCAL MACHINE (HKLM) 
HKEY CURRENT CONFIG (HKCC) 

Spoofed Keys are stored in the registry database under HKEY CURRENT USER. This 
portion of the registry database is set up when a user logs on and is stored back into the 
HKEY USERS when the user logs off. 

The App install block uses a .reg file. The keys in the .reg file are added to the registry 
by the regedit program provides. 

The structure of the keys will be as follows. 

HKEYCURRENTUSER 
Estream 

Registry Spoof 
Add 

HKCR 

HKLM 

HKCC 

Remove 

HKCR 

HKLM 

HKCC 

The eStream Registry spoofer needs to be able to delete keys as well as overwrite keys 
and add keys. The Overwrite and create key functions are provided by the Add Sub 
Keys. The Remove key may be a very sparse tree, but some software units will probably 
need the facility. 



Components 



Spoof Device Driver 

This is a device driver that has a DEVICEIOCTL command to toggle registry spoofing 
on and off 

Spoof Regedit Utility Program 

The utility program Regedit is not aware that registry spoofing is going on. It would be 
very helpful if an enhanced version of Regedit was developed that will be aware of 
registry spoofing and will identify keys that are added, deleted, or modified by the 
registry spoofing. 

eStreamControl 

This is an executable program that is started when the user logs on. The function of this 
program is to start up the eStream services that are required by the user to run eStream 
Applications. This program also shuts down eStream application services when the user 
logs out. 



Interface definitions 

Registry Spoofing uses the same interfaces that the registry uses. The only interface that 
is required is a device driver call to toggle registry spoofing on and off. The actual 
registry spoofer in the Windows Kernel will go to HKEY_CURRENT_USER\eStream to 
find its spoof database. 

StartRegSpoofingQ 

Causes the registry spoofer to begin spoofing using the current database. 



StopRegSpoofingQ 



Component design 

The registry spoofing device driver will intercept all of the following kernel level registry 
calls. 

RtlCheckRegistryKey 

NT STATUS 

RtlCheckRegistryKey ( 
XN ULONG RelativeTo , 
IN PWSTR Path 
) ; 

RtlCheckRegistryKey checks for the existence of a given named key in the registry. 

RtlCreateRegistryKey 

NTSTATUS 

RtlCreateRegistryKey ( 
IN ULONG RelativeTo , 
IN PWSTR Path 
) ; 

RtlCreateRegistryKey adds a key object in the registry along a given relative path. 

RtlWriteRegistryValue 

NTSTATUS 

RtlWriteRegistryValue ( 
IN ULONG RelativeTo , 
IN PCWSTR Path , 
IN PCWSTR ValueName , 
IN ULONG ValueType , 
IN PVOID ValueData , 
IN ULONG ValueLength 
) ; 

RtlWriteRegistryValue writes caller-supplied data into the registry along the specified 
relative path at the given value name. 



RtlDeleteRegistryValue 

NT STATUS 

RtlDeleteRegistryValue ( 

IN ULONG RelativeTo , 

IN PCWSTR Path , 

IN PCWSTR ValueName 

); 

RtlDeleteRegistryValue removes the specified entry name and the associated values 
from the registry along the given relative path. 

NTSTATUS 

ZwCreateKey ( 

OUT P HANDLE KeyHandle , 

IN ACCESS_MASK DesiredAccess , 

IN POBJECTATTRIBUTBS Obj ec tA t tri bu tes , 

IN ULONG Titlelndex , 

IN PUNICODB STRING Class OPTIONAL , 

IN ULONG CreateOptions , 

OUT PULONG Disposi tion OPTIONAL 

); 



Path 

Specifies the registry path according to the RelativeTo value. If 
RTL REGISTRY HANDLE is set, Path is a handle to be used directly. If the 
path intercepts any spoofed registry path then the path will be redirected to 
HKE Y_CURRENT_USER\eStream . 

Rt IQuery Reg is try Va I u es 

NT STATUS 

■? 

RtlQueryRegistryValues ( 
IN ULONG RelativeTo , 
IN PCWSTR Path , 

IN PRTL_QUBRY_RKGISTRYJTABLB QueryTable , 

IN PVOID Context , 

IN PVOID Environment OPTIONAL 

); ~ ~ • ■ 

RtlQueryRegistryValues allows the caller to query several values from the registry 
subtree with a single call. 

All of these Kernel level Registry functions need to be intercepted. If the path variable 
intercepts any path inside HKEY_CURRENT_USER\estream then the call will be 
redirected to that key. 

Private Helper Function 

BOOL IsPathSpoofed([in] PCWSTR Path, [out] PCWSTR eStreamPath); 

This will return TRUE if the path intercepts an eStream Key, FALSE otherwise. If the 
function returns TRUE then the corrected Spoof path will be placed in the eStreamPath 
argument. 



Re-Registration of Objects 

Many application programs such as Office Applications will register themselves every 
time they execute on the client machine. The Registry Spoofer will need to intercept 
these writes and re-direct them to the HKEY__CURRENT_USER\eStream set of registry 
keys. 



Testing design 

Generating Reg Key files and then using the normal regedit program to determine if the 
keys are being returned correctly will test this unit. Adding and removing registry keys 
using a client program that checks the spoofed path to the key can test this component 
quickly. 

The registry-spoofing diver will need to be tested with the following Windows SDK 
Registry Functions. 



Registry Functions 

The following functions are used with the registry. 



Function 

RegCIoseKev 
RegConn ectRegistry 

RegCreateKevEx 

RegDeleteKev 

RegDeleteValue 

RegDisablePredefinedCache 



RegEnumKevEx 
RegEnum Value 
RegFlushKev 
RegGetKevSecuritv 



RegLoadKev 



RegNotifvChangeKevValue 
RegOpenCurrentUser 



Description 

Releases a handle to the specified registry key. 

Establishes a connection to a predefined registry 
handle on another computer. 

Creates the specified registry key. 

Deletes a subkey. 

Removes a named value from the specified registry 
key. 

Disables the predefined registry handle table of 
HKEY_CURRENT_USER for the specified 
process. 

Enumerates subkeys of the specified open registry 
key. 

Enumerates the values for the specified open 
registry key. 

Writes all the attributes of the specified open 
registry key into the registry. 

Retrieves a copy of the security descriptor 
protecting the specified open registry key. 
Creates a subkey under HKEY_USERS or 
HKEYJLOCALMACHINE and stores 
registration information from a specified file into 
that subkey. 

Notifies the caller about changes to the attributes or 
contents of a specified registry key. 
Retrieves a handle to the 

HKEY CURRENTJUSER key for the user the 
current thread is impersonating. 



RegOpenKeyEx 
RegOpenUserClassesRoot 



RegOverridePredefKey 



RegQuerylnfoKey 
RegQueryMultipIe Valu es 



RegQueryValueEx 



RegReplaceKev 

RegRestoreKey 

RegSaveKev 

RegSetKeySecuritv 
RegSetValueEx 

RegUnLoadKev 



Opens the specified registry key. 
Retrieves a handle to the 

HKEY_CLASSESJROOT key for the specified 
user. 

Maps a predefined registry key to a specified 
registry key. 

Retrieves information about the specified registry 
key. 

Retrieves the type and data for a list of value names 
associated with an open registry key. 

Retrieves the type and data for a specified value 
name associated with an open registry key. 

Replaces the file backing a registry key and all its 
subkeys with another file. 

Reads the registry information in a specified file 
and copies it over the specified key. 

Saves the specified key and all of its subkeys and 
values to a new file. 

Sets the security of an open registry key. 

Sets the data and type of a specified value under a 
registry key. 

Unloads the specified registry key and its subkeys 
from the registry. 



The following are the initialization-file functions. They retrieve information from and 
copy information to a system- or application-defined initialization file. These functions 
are provided only for compatibility with 16-bit versions of Windows. New applications 
should use the registry. 

Function Description 

GetPrivateProfilelnt Retrieves an integer associated with a key in the 

specified section of an initialization file. 

GetPrivateProfileSection Retrieves all the keys and values for the specified 

section of an initialization file. 

GetPrivateProfileSectionNames Retrieves the names of all sections in an 

initialization file. 

GetPrivateProfileStrinq Retrieves a string from the specified section in an 

initialization file. 

GetPrivateProfileStmct Retrieves the data associated with a key in the 

specified section of an initialization file. 
GetProfilelnt Retrieves an integer from a key in the specified 



section of the Win.ini file. 

GetProfileSection Retrieves all the keys and values for the specified 

section of the Win.ini file. 

GetProfileString Retrieves the string associated with a key in the 

specified section of the Win.ini file. 

WritePrivateProfileSection Replaces the keys and values for the specified 

section in an initialization file. 

WritePrivateProfileString Copies a string into the specified section of an 

initialization file. 

WritePrivateProfileStruct Copies data into a key in the specified section of an 

initialization file. 

WriteProfileSection Replaces the contents of the specified section in the 

Win.ini file with specified keys and values. 

WriteProfileString Copies a string into the specified section of the 

Win.ini file. 

Obsolete Functions 

These functions are provided only for compatibility with 16-bit versions of Windows. 

RegCreateKev 

RegEniimKev 

RegOpenKev 

RegQueryVaiue 

RegSetValue 

Unit testing plans 

Unit testing can be performed with a simple client program that reads and writes keys in 
Spoofed Key folders. 



Stress testing plans 

Normal Windows operation reads and writes keys in high volume. One possible stress 
test would be to load and unload keys from the HKEY CURRENT USER/eStream/ 
database while simultaneously reading and writing the corresponding sub key folders. 

Coverage testing plans 

Test adding and removing keys from each of the spoofed key sets. 



Add a key to each of the following points 



HKEYCURRENTUSER/eStream/Add/HKCR 
HKEY_CURRENT_USER/eStream/Add/HKLM 
HKEY_CURRENT_USER/eStream/Add/HKCC 

Read the keys back from their corresponding keys in the following points. 

HKEY_CLASSES_ROOT 

HKEY_LOCAL_MACHINE 

HKEYCURRENTCONFIG 

Key deletion will also need to be checked. To accomplish this a set of keys is added to 
each of the following key folders 

HKEYCLASSESJROOT 
HKEY_LOCAL_MACHINE 
HKE YCURRENTC ONFIG 



Keys are added to the corresponding Key Folders 



HKEY_CURRENT_USER/eStream/Remove/HKCR 
HKEYCURRENTJJSER/eStream/ Remove /HKLM 
HKEY_CURRENT_USER/eStream/ Remove /HKCC 

A client application reads the keys using the windows SDK functions 

RegLoadKey 
RegOpenKey 



Cross-component testing plans 



The install manager needs to install key sets using a .reg file. Using the regedit program 
can check this to see if the keys have been installed correctly. 



eStream 1.0 Client Ul Modules 
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Requirements 

Here is a list of the eStream Client UI requirements 

1 . Administration of the Application Subscription and installation process 

2. Cache and File system Management 

3. License Management 

4. User Account Log in 

5. Application Syncing 

Users running the eStream system need a simple system to subscribe and unsubscribe 
applications. The user interface that provides for this will come through the web 
browser. A browser plug in ActiveX control can provide a simple way for ASP web 
pages to interact with the user client system. 

The File system driver will be loaded automatically when the user logs on. The Cache 
manager provides efficient data streaming to the file system driver. The Cache Manager 
is a worker thread inside the Client UI program. 

License management provides tokens that will allow a user to access specific application 
files from an eStream server. The License Manager is a set of threads running inside the 
eStream Client program that acquires, renews, and releases access tokens as a subscribed 
application runs. 

In order to access application executable and data files from an eStream Server a user 
must log onto an ASP server. The client UI program will provide this account 
management. 

Users can access eStream from different computers at different times. Changes in 
Application subscription status synchronized between these different computers when the 
user logs on. 



Functionality 

The eStream Client UI module supports reporting eStream-specific error & informational 
messages to the client user and solicits replies when appropriate. It allows the eStream 
client user to view and change the list of applications currently installed on the client 
system and the list of ASP accounts currently known to the client system. 

The eStream client will have a tray icon that will allow the user to access administration 
and control functions. Tray icons are small icons on the right side of the desktop toolbar. 
These tray icons give the user the indication that the eStream client is running and allow 
the user to access administrative functions that infrequently used. 



Figure 1 Tray Icon 



Right clicking on the tray icon produces a pop up menu that would allow the eStream 
user to access the following functions. 

• Set the cache size and review cache performance and utilization 

• Log the user into and out of ASP Accounts 

• Set the proxy server IP address 

• Set the ASP server 

• Access the ASP web page and view accounting and utilization information. 

Normal eStream The App Install manager will update Start Menus and desktop shortcuts 
in a mariner identical to a native installation of the same application. The user does not 
interact with the eStream client when running subscribed applications. 





eStream Tray Icon 



Components 



eStream Client exe 

This program is located in the startup folder and will be run when the user logs onto a 
system with at least one eStream subscribed applications. 

• Log on the ASP 

• Start the file Spoofer 

• Start the EFS file system 

• Start the License Manager and the Cache Manager 

• Get run tokens from the LRM 

• Contact the ASP(s) and update changes in subscribed application sets 

When the initialization is complete the eStreamClient.exe program will wait for the user 
to log off the system. When the user logs off the eStream device drivers (registry and file 
spoofers) are halted, LRM tokens and released, and the eStreamClient.exe Program exits. 



Browser Plug in 

An OLE Automation client is the simplest way for a web page to access internal 
configuration parameters on a Microsoft Windows Client. An OLE automation Client 
will allow a Browser script, VB Program, or VC program to communicate directly with 
the Client executable program. 

For a browser Script the eStream Client executable could be controlled using an object 
tag. 

Here is an example of HTML code that uses and Object Tag to connect to an OLE 
Automation server. 



<OBJBCT ID="ESTREAMCLIENT" 

CLASSID="CLSID:D8D77E03-712A-11CF-8C70-00006E127B7'' 
CQDEBASB= ht tp ; //www. aspcorp. com/eS treamClient . exe 
DATA=:"InstallSetASPOf ficel.ODS"? 
</OBJECT> 



Interfaces provided by the eStream Browser Plug In 



Applnstall (String AppInstallBlockPATH) 

The App install block contains a header section, variable section, and other blocks. The 
App Install block is large so a file link is sent to the Applnstall interface where the 
Applnstall block has been installed. This will probably be in some temporary folder. 

The Applnstall function will need to build an Uninstall block that will list the 
components of the subscribed application. Uninstalling applications cleanly and reliably 
is as important as important as installing them. The behavior of the App install manager 
should mimic the Add/Remove panel from the Windows Control panel. 

AppUninstall(AppId) 

Uninstall the application. The header file of the app install block that comes with the 
application when it was originally subscribed provides the Appld. 

XML data islands working together with Active Server Pages would provide a simple 
and reliable system for administrating Applications. 

Application Database 

A database containing application subscription information needs to be maintained on the 
client. This database can be displayed back to the user using XML tags. This database 
needs to contain enough information to provide the following functions the complete 
uninstall of subscribed installation, accounting information, and access to ASP accounts. 



Design of the Client UI Program 

The eStreamClient program provides all of the run time functionality that the client 
requires. The modules absorbed by this unit include 



• Client Log On 

• App Install Manager 

• License Subscription Manager 

• Client Network Manager 

• Cache Manager 

The App Install manager works in conjunction with the ASP web page to allow users to 
subscribe and un-subscribe applications. The coordination between the browser plug-in 
and the App Install Manager will probably require some kind of a COM interface into the 
eStreamClient program. 



The License Subscription Manager manages access tokens. Access Tokens are generated 
by the eStream Server for a limited period of time and must be renewed when they 
expire. The eStream Client UI has a set of threads that deal with this process. 

The Client Network Manager is the portal through which all communication between 
cache manager, license subscription manager, and the eStream server flows. 



The Cache Manager together with the file system driver is the core component of the 
eStream product. These two components serve application executables over the network 
to mimic native installation of application programs. 



eStream 1.0 eStream Client Network Component 
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Functionality 

The client network component communicates with the following servers for the types of 
requests listed. 

Account server 

Validate a user for this ASP and get subscription information 
DRM server 

1 . Validate a license for a subscribed app 
App server 

1 . Get an app install block for a subscribed app 

2. Open a file/directory for a subscribed app 

3. Various file requests on a previously opened file/directory 



Data type definitions 

Definitions provided here are only for data types that are shared among components; this 
will take coordination with these other components. Definitions should be as C-language 
structures, regardless of how they'll be implemented. 

Interface definitions 

Interfaces 

Only a few file operation interfaces are listed 

ValidateUserQ 

Input: 

• account server to query 

• user data to send, including perhaps 

o ASP identifier 



o username 

o password 

o client certificate 

Output: 

• subscription info for this user, including perhaps 

o currently subscribed apps 

o serial number for each app 

o DRM server to use for validation for each 

ValidateLicenseQ 

Input: 

• DRM server to query 

• subscription serial number to validate 

• client certificate 

Output: 

• access token for this licensed subscription 

• app server to use for data requests 

GetlnstallBlockQ 
Input: 

• app server to query 

• subscription serial number 

• client certificate 

Output: 

• app install block for this subscribed app 
AppOpenFileQ 

Input: 

• app server to query 

• access token (possibly NULL) 

• file name 



Output: 



• handle to use for further file requests 
AppReadFileQ 

Input: 

• app server to query 

• access token 

• file handle 

• buffer to fill 

• offset 

• length 

Output: 

. filled buffer 

• number of bytes read 

UploadAppProfileDataRequestQ 
Input: 

• account server to upload to 

• app profile data 

Output: 

• success/failure 

Component design 
Testing design 
Unit testing plans 
Stress testing plans 
Coverage testing plans 
Cross-component testing plans 



eStream Registry Spoofer current status 



Introduction 

Discussions about the need for active registry spoofing using a kernel level device driver 
and a complex database are presented here. There is a strong desire by the client design 
group to simply the design of the eStream software. 

Reasons for Registry Spoofing 

1. Leaving the registry pristine 

2. Allowing multiple users to have different views of the same computer 

One of the design goals of eStream is to provide a system of supplying applications to a 
user while having a minimal impact on that users system. The normal process of 
installing application software on a computer will make a large number of changes to the 
registry database on the client computer. One possible advantage of registry spoofing is 
keeping the client computers registry unmodified as applications are subscribed and un- 
subscribed. 

Another advantage of registry spoofing is to allow different users of the same computer 
to have access to different versions of software packages. This could be an advantage in 
situations where individuals share computers and eStream applications subscriptions are 
granted to individual users not to all the users of a particular computer. Two different 
users could use differing versions of the same application. A single user who does not 
subscribe to eStream could use one version and another user through an eStream 
application subscription could use a more current version of the same application. 

Reasons for not doing Registry Spoofing 

1 . Increased complexity of the eStream client software 

2. Difficulty of testing 

3. Adding complexity to the app install process. 

The difficulty of creating a kernel level registry spoofer is well understood by the 
members of the engineering team that have worked on Windows device drivers. The 
Registry is a component of the operating system that is critical to the proper functioning 
of both the Windows operating system kernel and most application software. It is 
accessed frequently by both the kernel and application software and any malfunction in 
its operation would have devastating consequences to the reliability of the eStream 
product. There are over 33 Windows SDK functions that access the registry directly and 



probably hundreds more that access it indirectly. Testing all of these functions would be 
a very difficult undertaking and would be critical to the success of the eStream product. 

In addition to the complexity added by a kernel level registry spoofer the addition of a set 
of spoofed registry entries for each subscribed application would add additional 
overhead, for both testing and installation, for each application that is available through 
the eStream subscription process. 

Other Spoofing Options 

1 . Normal Installation of eStream Subscribed Apps 

2. Logon time manipulation of the registry database 

Possible alternatives to spoofing the registry database include keeping the registry entries 
of subscribed applications in a separate database that would be added to the registry when 
the eStream client is started, when the user logs on, or at system boot time. These 
registry entries would be removed at logout or shutdown time. 

The popular consensus is to no perform registry spoofing at all on the eStream Client. 
The major advantage of this approach is to simplify the client application subscription 
process. The normal application install program that comes with each application could 
be used to subscribe an application if some way could be found to force the installer to 
place the application executables and portable data files on the EFS file system. 

Conclusions 

We have decided to abandon registry spoofing. 
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Functionality 

The Installation Manager consists of the following sub-components 
Installshield 

Installshield is the industry standard for building installation sets for Microsoft Windows. 
Installshield will take a set of executables and data files and create a media installation. 
The Installshield environment provides a scripting language that will allow a high degree 
of customization of target installation. The essentials issues for any installation are. 

1 . How much of the application does the user wish to install? 

2. Is the users system capable of running the application? 

3. Where does the user wish to install the application? 

4. Does the user have enough space to install the applications? 

Installshield has a wizard that will set up a project. When the install shield program is 
compiled a media must be specified. The most common media types are floppy, CD 
Rom, and Web media builds. For eStream we may have to ask the clients to reboot the 
machine since we are installing kernel mode components that might need a reboot to take 
effect. 

Install From the Web 

This program is another product that is sold by Installshield that will take a complete 
installation set and create a single executable .exe that can be easily downloaded from a 
web site. 

Uninstaller 

Installshield will provide an uninstaller when it builds the install program. 
Registry Settings 

There are three ways that Installshield can patch the system registry. 

1. Run regsvr32.exe on self-registering .dll files. When the uninstaller is run it will 
use regsvr32.exe /u to un-register the .dll file. 

2. Patch the registry statically. 

3. Patch the registry based on Installation Options from the install shield script 
program. 



Artwork 



The Installshield program for eStream will require a splash screen and possibly one or 
two other artwork components. 



Data type definitions 

The data that the Installation manager uses 

1 . Device Drivers 

2. COM Libararies 

3. COM Executables 

4. Registry files 

Interface definitions 
Testing design 

Testing of the eStream Installer must take place on computers when Visual Studio has not 
ever been installed. 

Unit testing plans 

Install the eStream Client using the binary file installer from InstallShield. Since the 
number of installation options will be kept to a minimum this test should not take very 
long. 

Installation of eStream Binaries from the InstallShield Installer is the first step of testing 
all new revisions of eStream. 

Stress testing plans 

The installation should be tested on a wide variety of computers with special emphasis on 
testing the installation on computers that do not have Visual Studio installed on them. 

Coverage testing plans 

Need component list for generating this test plan 

Cross-component testing plans 

All client components are installed with this piece so we don't have too much to worry 
about here. 
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Requirements 

The purpose of the License Subscription manager is to acquire and manage list of tokens 
that will allow the cache manager to access application executable and data files on 
remote eStream servers. 

Functionality 

This component is a set of worker threads that are part of the Client UI process. 

The LSM manages the users subscriptions to the different ASP accounts. It is part of the 
client component downloaded on a client machine. The LSM starts running when the 
client component starts running and is always active when the client component is 
running. Users on a given machine establish a connection with the ASP account servers 
from which they have subscribed applications. Users can add and delete the applications 
that are subscribed from the ASP accounts. The LSM makes the appropriate calls to the 
account servers to perform those actions. It gets serial numbers for the applications that 
are being subscribed and deletes them for the applications being un-subscribed (which are 
all part of the ASP ID Block). When the users start running any of the subscribed 
eStream applications, the eStream file system first queries the LSM before servicing any 
requests. The LSM in turn gets the appropriate access tokens from DRM servers along 
with the identities of application servers that can be used to run the applications. It uses 
the client identification (serial number) obtained when the connection to the ASP was 
made. At the same time, the LSM can decide to cache the access tokens and the identities 
of the application servers and decide to serve them directly from its cache. The eStream 
Cache Manager informs the LSM when applications start and end. The LSM keeps track 
of when access tokens are expiring and can request for additional access tokens when 
applications are running and the current one is about to expire. 



Component design 



The License Manger communicates with four other logical units inside the eStream 
Client. 



eStream Client Block Diagram 
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Figure 1 LSM interfaces 

This component may be a COM server component. We may decide to implement some 
of the functions of this unit as an in process DLL that will be access though COM 
interfaces. 



The License Manager communicates with four other logical units in the eStream Client. 
The interface with the Client UI control panel is through the 
ILicenseSubscritpionManager. This interface provides a complete list of all ASP 
accounts, subscribed applications, and accounting information to the Client Ul control 
panel. 



The interface with the App Install manager provides lists of application files when a new 
application is subscribed. These lists are stored in a database table. When an application 
is started access tokens are requested for the files that are part of the subscribed 
application. 

The interface with the client network provides a connection to the eStream server that 
will supply the application file binaries to the eStream Client. The function of the LSM 
is to request lists of access tokens. 

Threading Model 

In order to service token requests and present application subscription information to the 
Client UI in a timely manner the License Subscription Manager will need to make use of 
multi-threading. Currently three threads are planed to fulfill the design requirement of 
this component. The main thread will satisfy command requests from the Client UI and 
Cache Manager, and App Install Manager. A separate thread will be spawned when the 
License Subscription Manager starts to handle Access Token Renewal. A new thread 
will start for every access token requested or renewed by the Cache Manager. 



Main Thread 

{ 

Service Loop 

> 



Token Renewal Thread 

{ 

CKeck for Token. 
Rejvual 

Spawn Token 
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Token Request Thread 



™ Token Request Thread 



Token Request Thread 



Figure 2 LSM Threading Design 

The threads that provide License Subscription services will use Win32 SDK Event 
semaphores to signal to each other event notifications such a token renewal, network 
timeouts and token denial. 

Main Thread 

The main thread provides the interface support for the ILicenseSubscritpionManager and 
ITokenManager interfaces. When the main thread begins a worker thread is started that 
clears the token table by releasing and tokens that remain from the last instance of the 
License Subscription Manager. The token renewal thread sleeps on a timer waiting for 
an Access token to reach expiration. 



/* 

This is a psuedo code example of how the LSM main Loop will look like. The 
complexity of a COM implementation of this program unit means that the real code will 
look very different from this code. 
*/ 

LSMMainThreadQ 
{ 

ClearJTokenJJstO; 

Start_Token_Renewal_ThreadO; 
For(;;) 

{ 

WaitJForCommandO; 
Switch(CommandType) 

{ 

Case SubscriptionManagerRequest: 
ServiceJRequestO; 

Break; 

Case AccessTokenRequest: 
Get_Access_Token(); 

Break; 

Case AccessTokenGranted: 

FireAccessTokenGrantedO; 

Break; 

Case AccessTokenDenied: 

MessageBox(No Access Token); 
FireAccessTokenDeniedO; 

Break; 

Case AccessTokenExpired: 

MessageBox(Expired Access Token); 
FireAccessTokenExpiredO; 

Break; 

Case Shutdown: 

Release_Access_TokensO; 

Terminate_Token_Renwal_ThreadO; 

Return; 



} 

} 

} 



Token denial Policy and token expiration policy are two of the most critical issues that 
the License Subscription manager must handle. The policy for a token denial is to prevent 



the user from running the subscribed application. The policy for token expiration is more 
difficult. Currently the plan is to nag the user into renewing their expired subscription 
using message boxes. We may move to some other policy as the License Manager 
develops. 

Token Renewal Thread 

The token renewal thread is responsible for maintaining the current list of tokens and 
requesting renewal for each token as it expires. Each time a token expires a new Token 
Request Thread is started to access the Cline Network Interface for a new Access token 
from the eStream Server. 

TokenRequestThreadO 
{ 

InitiallizeTokenTableO; 

For(;;) 
{ 

WaitForMultipleEventsO; 
Switch(Event) 

{ 

case TimerPop: 

CheckForExpiredTokenO; 
For(each expired token) 
{ 

SpawnTokenRequestThread0; 

} 

Break; 

case TokenRequest: 

SpawnTokenRequestThreadO; 

Break; 

case TokenGranted: 

SetTokenGrantedEventO; 

Break; 

case TokenExpired: 

SetTokenExpiredEventO; 

Break; 

case TokenRefiised; 

SetTokenRefiisedEvent(); 

Break; 



Case Shutdown: 



Kill__Network_ThreadsO; 
Release Tokens(); 
CIean_Up_TokenTabIe(); 
EndThread(); 

Break; 



} 

} 

} 

Token Request Thread(s) 

A token request is spawned by the Token Renewal thread and it runs until one of the 
following conditions is met. 

1 . The network client performs a timeout. 

2. The Access Token is granted 

3 . The Access Token is refused. 

A class pointer passed to the thread from the LP ARAM function argument provides the 
actual token that the thread is requesting. 

UINT TokenRequestThreadProc( LPVOID pParam ) 

CRequestToken* pToken = (CRequest Token *)pParam; 

if (pObject == NULL | | 

!pObject->IsKindOf (RUNTIMEJ2LASS (CMyObject) ) ) 
return 1; // if pObject is not valid 

// Establish a connection with the Client Network Interfaces. 

IServerTokenManager . CreateDispatch ( ) 
IserverTokenManager .GetToken () 

If (Timeout) 

SignalTimeOut () 
Else if (Granted) 

SignalGrantedO 
Else if (Refused) 

SignalRefusedO 



return 0; // thread completed successfully 
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eStream App Server Low Level Design 

Version 1.2 
Sameer Panwar 



Functionality 



First, some definitions: 

eStreampage: the smallest unit of data that can be requested by a client from an App 
Server. Proposed to be 4kB for eStream 1 .0. 

page set: simply, a sorted list of eStream pages, each identified by a File ID (i.e. AppID 
& File #) and page # (essentially an offset into the file). This set is restricted only in that 
all pages in the set must have the same AppID. 

client request: a single self-contained message from a client requesting a page set from 
the server. Each server response to a client request can return a number of pages, and 
there is a maximum number of pages that the client can request in this message. (TBD, 
somewhere between 8 and 20 or so). 

The primary job of the App Server is to service client requests for application data blocks. 
The App Server is designed to minimize the amount of CPU time it must consume to sat- 
isfy each client request, thereby maximizing scalability. Thus, authentication is per- 
formed by a simple expiration time check of an AccessToken provided by the client, and 
compressed application data is saved persistently. 

The App Server serves data derived from eStream Sets. To decouple the performance 
needs of the App Server from the Builder, we should have a post-processing tool that 
converts the flat, uncompressed eStream Sets as provided by the Builder into a precom- 
pressed format suitable for memory mapping, if the App Server is configured to serve 
compressed bits. Also, a profiling part of the App Server can be used to monitor for 
common page sets, and then assemble more optimized replies, which compress the set of 
pages together as a unit, to take advantage of improved compression ratios. These replies 
can be stored on disk to save time in rebuilding them each time the server is started up. 

The App Server (AS henceforth) views an eStream Set as simply a set of files, and knows 
no further underlying structure. Thus an eStream Set contains at the start a table (FOST) 
indexed by File #, and providing the offset into the eStream Set where the associated file 
data begins, and the size of the file. So the AS just takes the client request of (AppID, 
File #, Page #, no. of pages), maps AppID to an eStream Set and looks up in the FOST 
table (File/Offset/Size Table) to find the requested data. 
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This works slightly differently when the eStream Set file has been pre-compressed by the 
post-processing tool. The resulting image is the same as before, except now the FOST 
points to another table, the POST (Page/Offset/Size Table). Because the compressed 
pages will be of different sizes, this table must by indexed by the Page # to find the rela- 
tive offset and size of the compressed page data for the file. Thus if an AS is not config- 
ured for data compression, the main difference in behavior is that it doesn't do a POST 
lookup and it doesn't care about coalescing page sequences. 

Data type & Data structure definitions 

Processed eStream set - this structure is kept on disk and never changes after installation 
It looks like: 

struct { 

Applicat ionID appID; /* for reference, is a 128-bit GUID, see ECM 

LLD */ 

uint32 maxFileNo; 

boolean compressed_f lag ; /* indicates whether the AppFiles are com- 
pressed, though maybe we should do it differently? */ 
FOST_Entry FOST [<maxFileNo>] ; 

uint8 appData[<sum of all AppFile sizes, which are variablol • 
} ProcessedEstreamSet; 

Since the files in the application are of variable size, we can't make a table out of them 
and must indirect out of a table (indexed by the File #) to find their offset location inside 
the AppData buffer. 

struct { 

uint32 offset; 

uint32 size; 
} FOSTJBntry; 

When the processed eStream set is compressed, then we use the AppFileCompressed 
structure at the offset indicated by the FOST, otherwise we interpret the data as just 
AppFile. The AppFileCompressed structure starts with a table that indicates the size and 
offset of the compressed data that belong to the page it was indexed by. 
struct { 

uint8 fileData[<size from FOST entrv>] 
} AppFile; ~ 

struct { 

POST_Entry POST[<number of pages, derived from size from FOST_Entry>] ; 
uint8 fileData[<sum of all FilePage sizes, which are variablol ; 
j AppFileCompressed; 

struct { 

uint32 offset; 
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uint32 size; 
} POST_Entry; 

This covers all the structures that live on disk. When we mmap-per-file, that means we 
make multiple mappings out of a single ProcessedEstreamSet file, at different offsets, one 
for each file. 

Now, for the in-memory data structures (assuming per-file-mmapping): 

The primary lookup will be a hash table, hashed on the AppID and FileNo. It should 
have on the order of 10,000 entries, each table entry containing a list of entries (for colli- 
sions). Each list entry contains: 

struct { 

ApplicationID appID; 
uint32 fileNo; 

uint 3 2 si ze ; /* size of the mapped Appfile */ 
MMap fileMap; 
HTLi st Entry * next; 
} HTLi st Entry; 

The Mmap struct just contains any OS-specific-related stuff to manage the mappings, 
plus a field char * pt r, which points to the place in memory that the AppFile (or 
AppFileCompressed) is mapped. So the hash table looks like: 
struct { 

HTListEntry * entry [<size of hash table>] ; 
} MMapHT; 

Hash function is TBD. The hash table should be statically sized large enough to handle 
the full number of eStream sets up to the maximum memory we will support. Assuming 
32 bytes being used per entry, that implies about 1 MB to handle 30k files, which is no 
problem. (Maybe we should reserve entries for 100k files or more?) 

Configuration : Each AS must obtain configuration data, either directly from the data- 
base or from the monitor in its startup message. The required data is (with the config pa- 
ram names and datatypes): 

AppList vector of ApplicationID 7 s (128-bit GUIDs) 

ServerPort uintl6 
MonitorPort uintl6 

SLiMKey uint (size TBD, depends on actual algorithm) 

ClientTimeOut uint32 
CompressionFlag uint32 



Network communication : The AS talks only to clients and the server monitor via the 
network. The server monitor communication will be described as part of the monitor 
heartbeat protocol. The AS-client communication will be described in a separate docu- 
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ment. The AS will time-out and close connections that have been idle for some amount 
of time (a few seconds). 

[maybe combine multiple responses into a single send socket call (will only work for 
TCP probably, since proxies won't like multiple server responses)?] 



Interface definitions 

The AS is optimized to do one thing only: serve pages from the read-only file system part 
of eStream, so there is just one interface with the client. Anything the client can care 
about in an eStream set is just another file to the AS, including the AppIrtstallBlock, and 
directories/metadata. The AS only returns the data the client requested, nothing extra. 

struct { 

uint32 fileNo; 

uint32 pageNo ; 
} PageRequest ; 

struct { 

uint32 errorCode; 
uint32 compressedFlag; 
uint32 fileNo; 
uint32 pageNo ; 

uint32 offset; /* offset into pageData below */ 
uint32 dataSize; 
} PageReply; 

PageReadRequest 

Caller: Client 
Callee: AppServer 

Input: uint32 appld; 

eStreamAccessToken accessToken; 
uint32 numPagesRequested; 
PageRequest pageSet[(numPagesRequested)]; 

Output: uint32 numPagesRequested; 

PageReply pageSetReply[(numPagesRequested)]; 

uint 8 pageData[(sum of all page data)]; 

uint32 globalErrorCode; 

Global Errors: INVALID_ACCESS_TOKEN 
EXPIRED ACCESS TOKEN 
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INVALED_APP_ID 

EXCEEDEDJVIAXREQUESTABLEPAGES 

Errors within 

PageReply: INVALID FILE NO 
INVALEDPAGENO 

SERVER_ERROR (probably should be logged, and should cause an alert if too many 
occur in some time period, including errors that don't get returned to the client.) 

AppServers don't ever talk to the database (it would be a waste of licenses considering 
the number of AppServers we'd have and their infrequent accesses). Instead, they obtain 
all their relevant control information from the server monitor. 

The exact interfaces are TBD, but from the monitor they will provide configuration in- 
formation, AppServer state change requests, and add/remove requests to the list of apps 
being served. Going back from the AppServer to the monitor, it will report load (average 
response time) on a per app basis, and server state, along with the heartbeat. 



Component design 

Interesting issues to deal with: 
Scalability/Performance 

Since scalability (and thus performance) is critical for the AS, let's go over how CPU and 
memory are used. 

Memory 

Performance is maximized when virtually all client requests can be satisfied by retrieving 
the desired pages from RAM, because RAM is far faster than disk. Thus the amount of 
RAM available will put an upper bound on the number of apps that a single AS can serve 
efficiently. Since server RAM won't grow as fast as the total size of all apps available as 
eStream sets, this means we'll have to heterogenize servers, where each server specializes 
in a subset of apps, limited by available RAM. For eStream 1 .0, this component of AS 
configuration will be handled manually, the eStream administrator assigning apps to 
servers. In the future, the set of App Servers should automatically reassign apps dynami- 
cally to balance load. 

But this is just one level of memory, committing RAM to a set of apps. There still re- 
mains the question of how to best utilize that RAM for each app, since some files are 
used far more often than others. This immediately means that for efficiency we must 
overcommit RAM, because if we allocate an entire eStream set into RAM, we're using 
precious resource to hold data that may be requested only very rarely. Instead of having 
to manage our physical RAM manually to accomplish this (such as with a cache), an eas- 
ier approach would be to take advantage of virtual memory (VM) to automatically keep 
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the hot pages in RAM, with the remainder available (again automatically) off disk (via 
memory mapping the eStream sets). That way the server can satisfy any possible client 
request for any app it serves, but is optimized to be the most efficient over all clients. But 
this only works if enough VM is available. (Time for some back-of-the-envelope num- 
bers.) Given that an app seems to have something like only 20% of it being hot (from our 
current limited data from the prototype), this means VM must be at least 5x of RAM for 
maximum efficiency. Given that a process has about 2 GB addressable VM, this corre- 
sponds to about 400 MB of RAM. Beyond that size (which is not uncommon), we don't 
have enough VM to efficiently overcommit our memory (by mmaping entire eStream 
sets). So now our choice is to either manually manage a memory cache (and all the atten- 
dant coding, bugs, etc.), or to mmap at a finer granularity. 

Note that the effective virtual memory required by an app is increased when compression 
is used, to handle the extra compressed page sets. They'll probably double or triple the 
RAM footprint by hot pages (due to redundancy), but only increase the overall VM foot- 
print by 1 .2 - 1 .5. The consequence of this is that the overcommit ratio goes down to 1 .5 
/ (3 * .2) = 2.5, though the amount of apps servable is reduced to 1/3 (! !). Now 2 GB vir- 
tual address space corresponds to 800 MB of RAM. This means we should be able to just 
memory map entire eStream sets, up to 2 GB worth, and be confident we're utilizing 
RAM efficiently, assuming the server has about 800 MB of RAM. A server with less 
RAM will likely thrash, and those with more will likely see little improvement in the 
number of apps they can serve via memory mapping. 

A loss of 2/3 in the number of apps an AS can serve I think is too great a sacrifice, too 
great a loss in app scalability (need 3x the number of servers as before!) for what is about 
a 1 5-30% greater effective bandwidth at the client. The root of this problem is the redun- 
dancy (costly in physical memory), because the compressed page sets will contain the 
same page in multiple sets. This is similar to the redundancy that appears in trace proces- 
sors and dynamic translation, which places extra memory demands in both those cases. I 
think we must completely eliminate this redundancy to achieve the goals we desire, either 
by (1) not using compressed page sets, and just sending multiple individually compressed 
pages, or (2) ensuring a page appears in only one compressed page set. [There further 
potential loss of effective memory size when using compressed page sets since they'll be 
allocated in 4k chunks, thus wasting about 2k on average; we'd have to batch them up 
together in files to minimize this. . . Also, saving the compressed page sets to disk intro- 
duces extra complexity to the AS because we'd have to properly handle recovery (i.e. 
what if the system crashes while we're writing the sets, which if we're memory mapping 
is totally out of our control). Because of this robustness requirement, and the fact we 
need to be 100% sure we're serving good bits (lest we crash a bunch of clients), this 
needs to be thought out very carefully if we want to do this. My opinion is that we should 
defer implementing compressed page sets until we better understand the tradeoffs, and 
good profiling schemes. In particular, will the AS be mostly bandwidth-limited, memory- 
limited or CPU-limited?] 

Separately from this, we should consider the effect of per-file memory mapping (ignore 
the compressed page sets now). This has the impact of requiring many more mmap's 
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from the OS, but promises better use of the limited virtual address space. In this scheme, 
we mmap each file into VM as it is referenced by a client. If only hot files are referenced, 
then the RAM footprint is the same as before, but VM is only used for the hot files, not 
the entire app, probably about 30-50% greater in size. Thus the overcommit ratio then 
becomes 1.5, much better than the 5 with full app mmapping. So 2 GB of VM corre- 
sponds to 1 .3 GB of RAM, much better than the 400 MB with full app mmapping. How- 
ever, this assumes that VM is used in a cache-like manner, evicting not recently used 
mmap's, since as uncommon files are referenced, they eat up more and more VM. Once 
VM is totally used up, then replacement policies and eviction (and fragmentation of vir- 
tual address space) become issues, just as with a manually managed cache. One solution 
is to simply purge all mmap's and start from scratch, which is simple and reliable, espe- 
cially considering the AS is multithreaded (if this is done, the above analysis doesn't 
hold, and performance becomes a function of how often VM is cleared). Another possi- 
bility might be to use the profiling mechanism and only place sufficiently popular files in 
mmaps and do regular file system accesses for the rest. 

Of course, the alternate option for managing physical memory is to know its size, and 
manage a cache manually. One advantage here is that the AS would know the physical 
memory consumption and usage (unlike when the OS was handling everything), which 
may help with load balancing. The main advantage is that there are no artificial limits 
(overcommit ratio is irrelevant), and only physical memory size is the true limit, and this 
approach can map any number of eStream sets (with any size of files) to any amount of 
physical memory up to the virtual address size (4 GB). Then memory management be- 
comes an issue (what do you do once all your RAM is full), which can be painful in a 
multithreaded environment. Again, we can just invalidate the whole cache as an option, 
but this will probably happen more often than with the per-file-mmapping case, unless 
RAM is greater than the maximum that the per-file-mmaping approach can handle. If the 
wholesale cleanup approach is used, then allocating fixed size chunks may not be needed, 
and we could potentially get better memory usage by packing compressed pages more 
tightly (e.g. 16-byte aligned vs. 4kB aligned), which is another potential advantage. 
Maybe instead of wholesale cleanup, we mark the most commonly used pages, and then 
just compact those and dump the rest (say 50%). The main issue with this approach is 
potential redundancy with respect to the OS disk cache (which is shared in the mmap ap- 
proach), and assumption that our caching policies will be better than the OS's. Also, 
lookups get messier, since we need a bigger lookup table to index via page # as well. 

Yet another option is to use multiple processes instead of multiple threads, one process 
per app being served, thereby releasing us from the 2 GB VM limitation. However, this 
introduces the issue of multiplexing requests from the network via IPC, and more load on 
the server monitor. On x86 NT, a Very Large Memory feature is available that can pro- 
vide 36-bit addressing per process; we may want to use this even though it won't be 
available on regular Unixes (and probably not x86-linux). 

In summary: per-eStream-set-mmapping is probably too wasteful of virtual address 
space. Per-file-mmapping is much better, but then memory management becomes an is- 
sue, suggesting a simple throw-away-and-start-over solution. However, given that solu- 
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tion, if a lot of physical memory is desired, a manual cache approach may be better (the 
better packing should overcome any loss due to redundancy with the OS disk cache). 
Compression of page sets invokes several issues that probably can't be fully addressed 
until after 1.0. Bottom line : the target now for 1.0 is to use p er-fi I e-m mapping with 
per-page compression (but no compression of page sets). Also, we should instrument 
the system to allow us to easily collect the relevant data (mem usage, CPU load of differ- 
ent routines, etc.) to help guide us in further evolution of the system to improve perform- 
ance (e.g. compressed page sets or explicit page cache). 

CPU 



The main work of the CPU is as follows (encryption is assumed to be done by hardware 
since its CPU impact is severe): 

1 • OS system call to retrieve request from network. 

2. Decode client request. 

3 . Validate AccessToken. 

4. Lookup AppID, File # in primary lookup hash table. (If mmapping eStream sets, in- 
stead lookup in App table, then lookup in FOST). 

5. If mmapping then (if uncompressed, no further lookup, if compressed, then lookup in 
POST to find page and size), if explicit cache then look in B-tree (secondary lookup). 

6. If lookup fails, then bring in the data off disk (either mmap or file system call). 

7. Copy page data to reply buffer. 

8. OS system call to send reply to network. 

However, if compressed page sets are used, lookups get more complicated, with a differ- 
ent set of tables to check for an appropriate page set first (and lookup failures incur poten- 
tial decompression/compression). It appears the least amount of CPU time is probably 
incurred when doing per-file-mmapping. All pages held in memory are kept in com- 
pressed form to save repeated compression of the same data, so pretty much all the work 
is in lookups and memory copies. Potentially the AccessToken validation will use hard- 
ware assist. Lookup failures (i.e. having to go to disk) should be relatively uncommon, 
and memory should be sized to ensure that. 

However, since the AS will run in user mode, this incurs the penalty of two extra copies 
(from the network buffers) and switching between kernel and user mode twice. If this is 
enough of a problem, we'll have to consider implementing the AS to run in the kernel (all 
commercial NFS, etc. implementations run in the kernel), which means we should choose 
our implementation to be compatible with that approach. In particular, we may not be 
able to rely on the virtual address space not being fragmented, so mmapping full eStream 
sets may be impossible. Plus robustness of the server becomes even more important, and 
portability issues arise. For the 1 .0 release, we plan to implement the AS in user mode 
keeping the possibility of moving to kernel mode in the future, and will collect data from 
1 .0 (or derived prototype) to evaluate the actual benefits. 

Disk : Since we are relying heavily on the common pages being in memory, we could 
possibly even consider storing the processed sets on a network disk, i.e. remote from the 
app server itself. However, such sharing won't work well for compressed page sets since 
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they are written to at runtime — it would be extremely messy to handle dozens of app 
servers trying to add many compressed page sets (possibly the same) to a set of shared 
files. 



Multithreading model 

The approach will be to have a single boss thread which pulls things out of the network 
port and stuffs client requests into a queue and a bunch of worker threads which grab re- 
quests and send back the replies. Simple enough, but this raises the issue of thread con- 
trol, since the boss also needs to be able to handle threads that die or hang and kill and 
restart them. The boss thread will monitor the worker threads and provide load/hearbeat 
info to the monitor through the server manager thread, thus giving visibility to the server 
monitor of the health of all the worker threads. 

Load balancing 

To be described elsewhere? (appears in SLiM server LLD) 



Security 

There are two levels of security involved in the AS. First, we must prevent clients who 
don't hold valid licenses from gaining access to the licensed binaries. This is accom- 
plished by the client obtaining an AccessToken from the SLiM server and presenting it to 
the AS upon every request. The AS can then use the SLiM server's public key to test the 
authenticity of the AccessToken (to protect against forgeries), and then can test the au- 
thentic expiration time of the AccessToken. Second, we must encrypt the actual data be- 
ing sent on the wire to prevent third parties from gathering the binary data covered by the 
license. Since the data coming out is somewhat obfuscated anyway (files are identified 
by arbitrary IDs, with our own strange message formats and compression and all in ran- 
dom pieces, etc.) it is not clear how much extra protection is really necessary, i.e. what do 
the license issuers actually want? We should use a common scheme like SSL to perform 
this encryption. It has been decided that the encryption load for this would be too great, 
and thus the data send back will be unencrypted. We may use SSL for authentication 
purposes only (i.e. null-cipher), if that is cheap enough. 

Also, a possible optimization for checking AccessTokens would be to cache recently used 
AccessTokens along with a signature/hash. If a token presented by a client matches, then 
we can skip the authentication step (since we've done it once already) and just check the 
expiration time. 
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Robustness 

The AS must be very robust. It must catch OS call errors and handle/log them as appro- 
priate, and deal with threads that hang or die. Thus it needs to aggressively check for er- 
ror conditions and possible failure modes. The AS also needs to track relevant resources 
(e.g. sockets, memory) and carefully manage/reclaim them so as not to exceed any limits 
or to degrade performance. And of course, the AS needs to check all data coming in from 
the client, to deal with ill-formed requests, and illegal values (e.g. huge negative indexes, 
etc.), and perform no potentially dangerous operation without validating parameters. This 
becomes even more important when we eventually move the AS to run in kernel mode. 
The AS also needs to be as stateless as possible, to minimize recovery time, and if it does 
perform writes to disk (such as for the compressed page sets), do so in a reliable fashion 
conducive to quick recovery. Any unreliability in the AppServer will negate any benefit 
of scalability we have over our competitors. 



Testing design 

This document must have a discussion of how the component is to be tested. Some sub- 
sections could include: 

Unit testing plans 

The various components of the AS are not too large or complicated: The request dis- 
patcher (to worker threads), the hash table, the compression code, the AccessToken 
checking code, etc. These shouldn't be too hard to do reasonable testing on in isolation. 

For the post-processor component, we'll have to build some sample Estream Sets as in- 
put, but it'll be hard to tell whether the output is correct without having a minimal work- 
ing AS. 

Cross-component testing plans 

The best approach will be to perform incremental implementation and testing. I.e. we 
build the core functionality that is required (i.e. can start with just regular i/o reads), and 
then add the more performance-related stuff later (adding mmaping, and then the hash 
table & AccessToken checks), while testing the entire system as pieces are gradually 
added (of course performing sanity-check and other minimal testing on the pieces first if 
possible). Compression can be added last. 

To actually drive the AS, we'll need a test client, which will be designed to just shoot off 
a series of read requests to the server. The file data returned could then be written to 
files, and this can be compared against the original set of files used to create the Estream 
Set we started with, to check that the data was received properly. For checking error con- 
ditions, a log of errors can be written and compared against a reference log for those re- 
quests we expect to fail. 
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Stress testine plans 

To accomplish this, we should run multiple independent test clients (on the same machine 
and on different machines), and increase the frequency of requests (to stress the AS's 
threads and sychronization, and communication routines), and the number and size of 
files referenced (to stress the hash table and memory). Each test client can then check 
whether the data and errors it got back were as expected, like in the above subsection. 

Coverage testing plans 

Should we use some kind of code coverage tool for this? 
Performance testing plans 

Since performance is critical, we should take the time to evaluate the AS's performance 
characteristics. We need to crank up our stress testing until either bandwidth or CPU 
saturates, and record the request rate that generated it. We should compare how this point 
responds to high numbers of clients with fewer requests per client vs. fewer clients with 
higher requests per client. We'll need to profile the system to find bottlenecks to tweak 
more performance out of it, and learn how well our original design assumptions hold up. 
Depending on whether CPU or bandwidth (or memory) saturates first, we may want to 
modify the system's tradeoffs to improve scalability further, and otherwise note which 
components a customer should upgrade for better performance. Also, if we think we can 
come up with reasonable client access pattern profiles, we may want to use those to esti- 
mate the actual number of real-world clients an AS can support. As part of this, we'll 
probably want to run the AS in-house once it is mature enough (eat our own dogfood), 
and then farm out app upgrades, etc. (play out some of our scenarios) and see what hap- 
pens to the AS's (do they choke or what). 

Availability testing plans 

We will also need to test our failover and load balancing capabilities. This will require 
several test machines with the monitor in place to start and stop servers, and have clients 
be aware of multiple AS's and respond appropriately when an AS stops responding. For 
load balancing, we'll probably want a bunch of test clients with a variety of access pat- 
terns and see how well their requests are distributed. 



Upgrading/Supportability/Deployment design 

App Servers will possibly need to version their interface with clients (requiring clients to 
state the version they're expecting), but will also need to support older versions. We may 
also modify the Estream Set format (or just the processed set format), but that should be 
handled by upgrading both the AS and post processor and then regenerating the processed 
sets. 
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For supportability & deployment, the AS will report error conditions and load to the 
server monitor, which is used by the customer. 



Open Issues 

1 . Is there a limit to the # of possible mmaps? 

2. Is there a single system call to unmap all mmaps? 
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Abstract: The following document presents a server framework based on 
CORBA for eStream 1.0. 

Descriptions: eStream 1.0 is a distributed server environment. CORBA provides 
a cross-platform cross-language distributed system solution. The high level 
essential features of the CORBA framework are listed below. 

• Messaging Support. How do the servers and clients, servers and servers 
talk with each other. CORBA provides mechanism for inter-object 
communication on a variety of protocols (HOP, GIOP, HOP over HTTP). 

• Distributed Object Management. This essentially is useful for management 
and monitoring in eStream as the client side objects eStream supports are 
fairly simple. However for management and monitoring all servers need to 
provide objects which advertise the health of the system. 

• Services: Corba provides a variety of services for a distributed system. 

o Naming Service. Helps maintain the location of objects in the 

systems. This is very useful for server management tools, 
o Event Sen/ice. Useful for Alarms etc. 

o ORB service. Used for server configuration, server state. It has the 

capability to stop/start servers, 
o Security service. Useful to access control and encryption services, 
o Distributed Transaction Support. Probably not relevant to our 

framework. 



The following diagram illustrates the eStream architecture at a very coarse level. 




Listed below are the objects in our system and the data they manipulate. 



CLIENT 


DATA 


LOCATION 


eStream Client 


Account/User/Subscription 
Information 


RDB/LDAP 




EStream Sats 


File System/RDB 




Server 

lnformation(Location of 
ADRM, APP, Profile etc) 


??? 


User/Account 
Management Client 


Account/User/Subscription 
Information 


RDB/LDAP 




Server 

lnformation(Location of 
ADRM, APP, Profile etc) 


??? 


Server Administrator 
Client 


Server 

lnformation(Location of 
ADRM, APP, Profile etc) 


??? 




Real time/Heart Beat 


??? 





status of the servers 






Load information for the 
servers 


??? 




Configuration information 
for servers 


??? 


Servers (as Clients) 


Static Configuration of 
other servers in the 
system. Give me a server 
which serves Word. 


??? 




Dynamic Configuration of 
other servers in the 
system. Heartbeat and the 
load are examples of this 
information. 


??? 




Load/Logging/Alarm 
Information. Log this 
access. Write down my 
load. Raise this alarm to 
the administrator. 


??? 



The question marks in the table above are transient data which characterizes the 
current state of the servers in the system. A CORBA based system will solve this 
problem using the following server architecture. 




The management client in this scenario will essentially talk to the CORBA system 
to get any information of the servers in our system. 

Listed below are the pros and the cons for a CORBA based system. 
PROS: 

1 . A well-defined and proven server framework. 

2. Cross platform support. 

3. Lot of services is available for free. Alarm, Management, Load Balancing. 

4. Distributed. The objects in the system are inherently distributed and hence 
more scalable. 

5. High performance system. Transient data about the system is stored in 
transient storage and hence data accesses are fast. 



6. Tools and services are available for free. Example: A distributed 
transaction support system is available and may be useful for eStream in 
the future. 

7. Server management and Alarm tools are easily available. 
CONS: 

1 . Vendor Lock in. Visigenix and lona are primary vendors with their own set 
of quirks. Both do not have a good history of migration support. (Partly due 
to the CORBA standard evolving very rapidly). 

2. In house expertise. 

3. The cost of the solution may be too high. (Need to investigate on this). 

4. May a very complicated solution for a simple problem. 



eStream 1.0 Database Centric Server Framework 
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The data centric Server Framework Architecture uses the database as the center 
of all communication channels. The following diagram illustrates the architecture 
of and eStream system based on the database framework. 



The following data model diagram illustrates the data model of a system 
management scenario in a database centric server framework. 



Log 

— Q— 



Load 

— O— 



-o Server 



Config ex- 



Machine 



Server: A logical server instance in the system. This will have the state attribute 
associated with it. The state attribute will describe the current state of the 
machine. 

Config: A set of configuration values for a server. The configuration values can 
be hierarchical. 

Machine: A physical machine. This will have a set of parameters describing the 

physical machine. IP, Physical memory etc. 

Log: This will maintain the log for eStream servers. 

Load: Used to record historical and real-time load on a logical server instance. 



The interactions with the database in this model are denoted below. 



1 . Read/Write Configuration. This is used to update the configuration table 
for a given server instance. Reading the configuration data will involve a 
simple select from the configuration table. 

2. Read/Write Machine Records. Create and update physical machine 
information. Not done frequently. 

3. Read/Write Log information. This is done once per heartbeat. A batching 
mechanism may be appropriate here. This is done once per heartbeat. A 
batching mechanism may be appropriate here. 

4. Read/Write Load information. This is done once per heartbeat. A batching 
mechanism may be appropriate here. This is done once per heartbeat. A 
batching mechanism may be appropriate here. 

5. Read/Write Server information. Done when creating new server 
configurations. It may be useful to duplicate some information here for 
faster access, (eg The most recent load characteristics.). 

Thoughts on installation of server components 

• All server side components including server side client interfaces with the 
exception of the database to be installed on every machine within a 
deployment for estream v1.0 (for future releases we can consider 
something more elaborate) 

• monitoring process will be started (the server client admin can launch this 
from its Ul), find and connect to database, configure itself, and then 
configure and launch logical servers. 

• server side clients can be launched independently of the monitoring 
process. 

o Server side clients can configure and read/write from the db even if 
the monitoring process and all logical servers are not running. 

Thoughts on Monitoring process 

• A monitoring process is required to monitor the state of the logical servers. 

• monitoring process is responsible for load balancing and maintaining high 
levels of QoS. 

• One monitoring process per deployment installation. 

o monitoring process can monitor across machines 

• monitoring process can re-launch a server side process (this can be a 
configurable option) 

• monitoring process can be used to start and stop server side processes 

• Monitor requests heart beat request for each logical server. If failure to 
respond in reasonable time, it will post an alarm in the database. 

• heart beat request rate is configurable. 



• heart beat request has a very simple and easy to decode protocol which 
may include: 

o simple request pulse 

o stop request pulse 

o dynamic config request pulse 

• Each pulse reply may include current load information from logical 
servers. 

o key servers to monitor load is DRM and App 

• Monitor needs to track load of DRM and APP minimally. 

Random thoughts on db updates and logging 

• log atypical events instead of typical when logging process state changes. 

• DRM (all server processes in general) to maintain performance critical 
information in memory as well as in db for quick response. This ensures 
persistent state if the DRM crashes yet hopefully allows for very fast 
response. 

• Assume that the performance requirements on the webserver are such 
that it can read/write to the db on every transaction if necessary. 

• Access token to maintain the app server for getting requests. Client 
needs to do quick check on app server to see if there is change. If so 
client needs to re-direct to different app server. 



Thoughts on configuration 

• Server configuration can only be performed by a server-side 
administrative client. 

• Typically, all configurations go through the db where they are persistently 
maintained. ' 

• administrative client changes causes table updates in the database. 

o client could send direct socket update to monitor process (if it is 
running; can be determined by client from db) for requests for 
dynamic configuration change. 

• monitoring process reads tables in the database for configuration 
information and will launch server side processes as appropriate. 

• Once a logical server is up an running, modifications to the configuration 
are either: 

o read directly from the db by the logical server and acted on (polling) 
o monitor sends special heart beat request which states that the 

logical server should recheck its configuration, (event driven) 
o monitor shuts down process and restarts to pick up config change 

through the normal start-up process 
o monitor communicates configuration through heart beat protocol. 



Issues to Resolve 



How would the monitoring process remotely start a server process. 

o once process is started monitor knows since it will respond to 
the heart beat request that will be sent after process is 
launched. 

How does monitor get launched. Monitor needs to get to the database 
and access with proper password. 

o more detailed config information for monitoring tool can be 
gathered from database. 

Need to come up with some concrete examples showing dynamic config 
update. 

Need to make distribution system proposal 
Ask Anne to determine what the data sets for profiles will look like, 
o will these profiles go into the data base or flat files. 



eStream Configuration Management Low Level Design 

Bhaven Avalani 



Functionality 

The configuration management utility is used by all server components to manage the 
server configurations. It provides the following functionality: 

• Configuration for a server consists of a set of name - value tuples where the val- 
ues themselves can be a set of name-value tuple. 

• Servers can load the complete configuration from the database. 

• Servers can load the configuration for a given name. 

• Server should be able to load the configuration from a flat file also. 

On the Server Manger interface, configuration will appear as a table containing name - 
value tuples. The table may be hierarchical to represent nested structures containing the 
values which can themselves be name values. An example of a simple name- value pair 
would be: 
PORT 8080 

An example of nested name values would be: 
Applications: 

word, exe windows2000sp3 

excel.exe win98sp4 
On a flat file the configurations will always be name-value pairs. To represent one level 
nested structure the format would be: 
Applications word.exe windows2000sp3 
Applications excel.exe win98sp4 



Data type definitions 

Class tuple { 

string name; 
Value value; 

}; 

class Value { 

int type; 

}; 

class StringValue: public Value { 
string value; 

}; 
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class TupleValue: public Value { 
vector <tuple> tupleArray; 

}; 

typedef vector < tuple > configArray; 

class ServerConfig { 
private: 

configArray Array; 

public: 

ServerConfigO; // Default initializer 

ServerConfig(string filename); // To initialize from a file. 

ServerConfig(ServerId, Dsn, Dbuser, Dbpasswd); // Initialize from DB 

configArray* GetConfigArray(int serverld); 

tuple FindConfig(int serverld , string name); 

Reload(int serverld); 

GetConfig(int serverld ,string name); 

}; ■ 



Interface definitions 

GetConfigArray 

configArray* GetConfigArray(int serverld); 
Inputs: 

int serverld 
Outputs: 

An array containing the configuration information. 

FindConflg 

tuple FindConfig(int serverld string name); 
Inputs: 

int serverld 

Name of the configuration to find. 
Outputs: 

The tuple containing the name and value for the config 

Reload 

void Reload(int serverld) 

Comments: 

Reload all the configuration for the server information from the database. 

GetConfig 
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void GetConfig(int serverld , string name); 
Comments: 

Reloads the configuration for a given named configuration. 

Component design 
Testing design 
Unit testing plans 

The configuration module is a stand-alone module which will be tested using a config- 
test.exe executable. The executable will exercise all of the interfaces described above. 
The configtest executable should be testable in the DB and the non-DB mode. 

Stress testing plans 

Not relevant to this component. 

Coverage testing plans 
Cross-component testing plans 

Upgrading/Supportability/Deployment design 
Open Issues 
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Functionality 

The eStream client will talk to the eStream servers using the HTTP protocol. This allows 
the eStream clients to run across a firewall. Since, the clients and the servers are both de- 
veloped by the OTI, we will implement the minimum HTTP protocol to optimally server 
our environment. The subset of the HTTP protocol we will implement is: 

HEADER 

• POST primitive. (GET/LOAD are not applicable to our situation as we will al- 
ways post our data structures in pre-defined format.). 

• Keep- Alive primitive. (Needed for maintaining the connections). 

• Host primitive. (Needed for maintaining the connections). 

• Content-length primitive. ( Needed to access the POST data). 

The process of client server communications over HTTP is explained in the diagram be- 
low. 



Request 
Decoder/ 
Encoder 



HTTP 
Enabler 



JN 



HTTP 



Client 



HTTP 
Listener 



Request 
Ecoder/ 
Decoder 



Server 



The following example shows the lifecycle of a request in this model. 

1 . Client requests for a file. GetFile(string fileDD, string version); 

2. Encoder encodes the request into a bitmap structure(To be defined). 

3. HTTP Enabler plops in the following header to the request: 

POST /HTTP 1.1 
Host: <servername> 
Conneciton: KeepAlive 
Content-length: <contentJength> 
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<content> 

4. Listener gets the request and parses the request get the content length etc. 

5. The content is forwarded to the Request Decoder. 

6. Response is generated on the server. 

7. Http Listener plops in the HTTP header to response, 

8. Client gets the response back. 

Based on the discussion above there are essentially three major components in this archi 
tecture: 

1 . Request Encoder/Decoder. Takes C type requests and encodes them to a binary 
format. 

2. HTTP Client: Makes simple HTTP 1.1 requests. 

3. HTTP Listener: Receives request, parses them and then forwards them to the de- 
coder. 



Data type definitions 
Interface definitions 

void sendMessage(char* mesg, char* ip, hit port = 80, char** reply) 

Inputs: 

mesg: The message buffer to be sent, 
ip: IP address of the server to send the message to. 
port: Port number to send the message to. Default is port 80. 
Outputs: 

reply: The reply from the server. 

Description: The HTTP client will make this request to send a message to the server. 

Errors: 

IO Exception. Occurs when the message is not deliverable, 

void readRequest(HTTPRequest* Req) 

Outputs: 

HTTP RequestStructure. 
Description: 

Called on the server side to read and parse the incoming request on server listener 

socket. 
Errors: 

INVALID JIEUQEST (We do not support GET and LOAD) 
IO Exception. Socket error on receive. 
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void sendResponse(char* mesg, HTTPRequest* Req) 

Inputs: 

mesg: The response message to be sent. 
Req: The Request structure containing the original request. 
Description: 

Called by the server to send a response to a request. 

Errors: 

IO Exception. Socket error on send. 



Component design 
Testing design 

This document must have a discussion of how the component is to be tested. Some sub- 
sections could include: 

Unit testing plans 
Stress testing plans 
Coverage testing plans 
Cross-component testing plans 
Upgrading/Supportability/Deployment design 

This document must have a discussion of how the component addresses any specific is- 
sues related to upgrading, supporting and deployment of e-stream applications. Some ex- 
amples include: error conditions detected and reported by this component, any special 
hooks this component will provide for monitoring, hints for troubleshooting problems, 
any special hooks for debugging this component. 

Open Issues 

This is a list of issues that need to be fiirther investigated or revisited during implementa- 
tion. 
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Functionality 

All servers and clients in eStream 1 .0 need to log the error and access data. Logging en- 
ables component debugging and auditing support. 

EStream Framework should provide logging with the following features: 

• Each component will have an error and optioanally an access log file. The names 
of these files would be <component>_error.log and <component>_access.log. 

• The files will be located in the <eStreaml .0 Root Dir>\logs directory. 

• The error log files will have messages with the following priorities: 

o 1 -Low : A warning which can be ignored. 

o 2-Medium: A warning which needs to be looked into. 

o 3-High: Recoverable error in the component. 

o 4-Critical: Irrecoverable error. Needs admin assistance. 

• Logging level should be configurable. The following levels are to be supported. 

o 0: Only errors will be logged. This will be the default level, 
o 1 : Errors and Warnings to be logged. 

o 2: Errors, Warnings and Debugging information to be logged, 
o 3 : Errors, Warnings and advanced Debugging (like memory dumps, tcp 
stack dumps etc) to be logged. 

• Log Wrapping to be supported. The log files will wrap at a predefined size. On 
wrapping the following actions will occur: 

o Any existing <logfile>.bak will be deleted from the system, 
o The current <logfile> will be backed to <logfile>.bak 
o The component will continue logging to the <logfile>. 



For each eStream client and server component logging the log files (component__error.log 
and component_access.log) should be written in eStream l.ORootUogs directory. The 
formats for the log files will be as follows: 

Error Log: 
[HEADER] 

[Thread ID] [TimeStamp] [Priority] [Message] 
[FOOTER] 
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An example of this log format would be: 



********************************************** 

Omnishift eStream Application Server 
Server Started. 

StartTime: i4/Aug/2ooo:i6:3i:i9 -0700 
IP Address: 1.1.1.1 
Logging Level: 3 

******************************* *************** 

0 [14/Aug/2000:16:31:19 -0700] 3-High Cannot connect to the database. 
Invalid Us ername/ Pas sword. 

1 [14/Aug/2000:16:31:19 -0700] 4 -Critical Cannot start the HTTP listener 
at port 80. 

0 [14/Aug/2000:16:31:19 -0700] 4-Critical Shutting down the server. 

******************* ++ ^^ + ^^^^^^^ ++:>:l5S|5:#:J|s:fs:|c+:|c:4c:ie:#E 

Omnishift eStream Application Server 
Server Stopped. 

StopTime: 14/Aug/2000:16:35:19 -0700 
IP Address: l.l.i.i 
Logging Level: 3 

********************************************** 



Access Log: 
[HEADER] 

[Thread ID] [TimeStamp] [Message] 
[FOOTER] 



Data type definitions 

typedef enum{l-Low, 2-Medium, 3-High, 4-Critical} ErrorLevel; 
typedef emim{0,l,2,3} LogLevel; 

Interface definitions 

SetErrorLogFile: Set the error Logfile. 
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Bool SetErrorLogFHe(string filename) 
Input: 

Filename: Name of the file. 
Output: 

Success/Failure 

Error: 

I/O FAILURE 

SetAccessLogFile: Set the error Logfile. 

Bool SetAccessLogFile(string filename) 
Input: 

Filename: Name of the file. 
Output: 

Success/Failure 

Error: 

I/O FAILURE 



SetErrorLogFileSize. Set the error log file maximum size, 
void SetErrorLogFiIeSize( int fsize = 10 ) 
Input: 

fsize Size of the file in MB. Default is 10. 
Comments: 

If this API is not invoked, then the file size defaults to 1 0 MB. 



SetAccessLogFileSize. Ser the access log file maximum size, 
void SetAccessLogFileSize( int fsize =10) 
Input: 

fsize Size of the file in MB. Default is 10. 
Comments: 

If this API is not invoked, then the file size defaults to 1 0 MB. 

SetErrorLogLeveL Set the log level of logging errors, 
void SetErrorLogLevel( enum LogLevel = 0) 
Input: 

Loglevel enum defined above. Default is 0. 
Comments: 

Can be called any time during the execution to change log level. 

LogError. Log an error message. This interface will take in variable arguments, 
void LogError(long threadid, enum ErrorLevel, char* format, ...) 
Input: 

Thread id if the caller thread. 
Error level. Enum defined above. 
Format. Printf like format. 
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Variable number of arguments. 

Errors: 

INTERNAL 10 ERROR 

LogMessage. Log an access message. 

void LogMessage(long tbreadid, char* format, ...) 
Input: 

Thread id if the caller thread. 
Format. Printf like format. 
Variable number of arguments. 

Errors: 

INTERNAL IO ERROR 



Component design 
Testing design 

This document must have a discussion of how the component is to be tested. Some sub- 
sections could include: 

Unit testing plans 

The logging utility will be built as a DLL (otlog.dll). We will provide a binary otlog- 
testxxe which will exercise each of the interfaces mentioned above. 

Stress testing plans 

Use the unit testing executable in a mode where the logging files are overflowed etc. 

Coverage testing plans 
Cross-component testing plans 

Upgrading/Supportability/Deployment design 
Open Issues 
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eStream 1.0 Low Level Design 
Software License And Management (SLiM) Server 

Am itPatel 
Last Modified: 

Version 2:0 

Functionality 

The Software License Management (SLiM) server is required to enforce licensing terms and 
track overall application usage. Its primary function is to grant, renew and expire access tokens, 
record application usage and aid in server load balancing. Its design can be broken down into 
three somewhat orthogonal axis: 

1 . Detailed specification of eStream client interfaces - the need. 

2. Design and usage of Server Common Services (CSC) & server database -the tools of the 
trade. These include logging, system monitoring, thread package, encryption, TCP/IP 
communications, etc. 

3 . Core SLiM server logic that fulfils client interfaces (1 ) using CSC (2). 

This document address items 1 and 3 in detail; item 2 should be covered in various other 
documents emerging from the server team. 



Data type definitions 

Common Data Types 

□ Standard atomic data types everyone (clients, builders, servers) must agree on: Int8, 
Intl6, Int32, Int64, ulnt8, ulntl6, ulnt32, ulnt64, ulntl28 (specially for GUIDs). 
o Notice: No floating point types; I don't see a compelling reason to pass floating- 
point number across wire. 
Q String is represented as a size (ulnt32) and it contents. Its contents must include a NULL 
termination character, it must be included as part of the size field. 



T .e.npth 



characters. . 



Q All eStream sets will use little endian representation. 

□ All complex data structures between major components (def: at least client/servers) 
should be version identifiable. Proposal: put a version number (uint32) as 1 st word in the 
structure. 

□ Most complex structures are variable length because they contain strings. I think it would 
be good to put the total size (in bytes) of that structure as second word so that reader can 
know how much to read. If marshalling/unmarshalling utilities provide a way to represent 
that, it won't be needed in each struct. 

□ We'll need a single place where all globally (definition: across client and servers, and 
between servers) visible macros (#define & enum) are defined. We'll also need reserved 
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name spaces, and reserved number ranges for different components. Until all that is 
decided, I am defining macros without assigning any values. 
□ I am assuming that our message pack/unpack utilities will deal with alignment issues. 

Server Sets 

EstreamServerlD contains server parameters clients need to know in order to initiate a 
connection. EStreamServerSet is simply a list of individual server Ids. The main use of this data 
structure is for SLiM server to return a list of app servers that can serve a given application. 
Server ids and server sets are specific to each application; client is responsible for keeping a map 
of app id -> server set. 

#define SERVER_TYPE_APPLICATION 
#define SERVER TYPE SLIM 
#define SERVER_TYPE_ASP_WEB 

typedef struct { 

ulnt32 Version; 

ulnt32 SizelnBytes; 

ulnt32 MachinelP; 

ulntlS MachinePort ; 

String MachineName ; 
} eStreamServerlD; 



example of an eStreamServerlD : 

(1,20, 0x12348, 80, (12, "slO.asp.com"}} 



typedef struct { 
ulnt32 
ulnt32 
ulnt32 
ulnt32 

eStreamServerlD 
} eStreamServerSet; 



Version, 
SizelnBytes, 
ServerType; 
ServerlDCount , 
Server ID [] ; 



example of an eStreamSexrverSet: 
{!,??, SERVER _TYPEJ\PPLICATION, 2 , 

{1,20,0x12348, 80, (12, "alO.asp.com"}}, 

{1,20, 0x12349,80, {12, "all.asp.com"}}} 

Access Tokens 

This is a main data structure that is getting passed back and forth between a client and SLiM/App 
servers. Granting an AccessToken is an acknowledgement of client's legal right to run the 
application - the license. Denying an AccessToken is an acknowledging that the client does not 
have rights to run the application; probable causes include a user running multiple sessions, user 
not paying bills etc. 
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From client's perspective, it is totally opaque; but SLiM server uses it to pass information to the 
app servers so that the app server does not have to rely on the database lookups. Each access 
token will have a unique JD. 

Terminology 

Billing Granularity - Granularity at which an ASP is interested in billing its customers. Most 
ASPs today bill on monthly basis and eStream will assume that to be the 'norm'. However, to 
support things like short trial memberships, we'll design eStream to handle billing as often the 
AccessToken Renewal Frequency (defined below). If an end user simply purchases the 
eStreamed application, the billing granularity is infinite - the upper bound. EStream should not 
assume that billing granularity for all apps served by an ASP is the same. 

AccessToken Renewal Frequency - Frequency at which the client must renew its access tokens 
in order to continue eStream application use. This must be tunable parameter whose upper bound 
is the billing granularity; it is also the smallest billing granularity we'll support. Not all access 
tokens are required to have the same renewal frequency. 
Recommendation: 10 minutes. 
Tradeoffs: 

1 . This is the smallest granularity at which a client can be evicted (defined below). 

2. Finer granularity may increase the number of hits to the SLiM server and adversely effect 
its scalability. 

Eviction Notice - In general there will be times when an ASP wants to stop a user from using an 
eStream application, which also means stopping a user from consuming ASP server resources. 
Possible reasons may be: 

• Lack of payment. 

• Termination of a trial membership. 

• To force the client into upgrading an app. 

• Just because the restroom is freezing cold. 

EStream infrastructure has an inherent limitation that servers can't push anything on the client. 
That means SLiM servers must deny an access token or its renewal, to effectively deliver an 
eviction notice to the client. Also, App servers may need to be informed of such evicted access 
tokens so that they can deny paging requests. 

Decision: After looking at some scalability numbers, we concluded that a renewal frequency of 
10 minutes should not affect the overall performance and scalability of eStream system. 
Consequently, we don V have to communicate the list of evicted tokens to the app server 
since they would be invalid soon (avg 5 minutes) anyways. This simplifies server designs 
by reducing cross communication between slim servers and app servers. 

typedef struct { 

ulnt32 Version; 

ulnt32 SizelnBytes; 

ulntl28 ATID; // AccessToken ID - GUID 

String Userld; // GUID. 
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ulntl28 AppID; // GUID. 

ulnt64 IssueTime; // POSIX time_t format. 

ulnt64 ExpirationTime; 
} eStreamAccessToken; 

Other Common Data Structures 

In order to allow easy/automatic updates of eStream application, we need to define a protocol by 
which a client can be informed of app updates. This structure will also be used when installing 
subscribed applications on a client. 

AppName, VersionName - describe the application. 

Message - a short description of an application. 

Flages, such as ForcedUpgrade - client must upgrade the application. 

RootFileNumber - is sort of the version # of an application root directory. 

RootFileMetadata - metadata of the root directory. 

typedef struct { 



ulnt32 


Version; 






ulnt32 


SizelnBytes; 






ulntl28 


AppID; 






String 


AppName ; / / 


may 


be "Word2000" 


String 


VersionName; // 


may 


be "SP1" 


String 


Message; 






ulnt8 


ForcedUpgrade ; 






Int32 


Roo t F i 1 eNumbe r ; 






???? 


RootFileMetadata; 







} eStreamAppInfo; 



Interface definitions 

In a single process context, cross-module interfaces are easy and intuitive when defined as 
C/C++ procedure calls. However, for client/server (and perhaps server/server) interfaces, we 
need to define our own RPC-like protocol. Sameer covering this (EMS - estream messaging 
services) in a different design, but I want to state couple of assumptions I am making: 

• Each EMS call is assigned a unique number (Int32). Codes must be uniform across all 
servers (i.e. no duplication of names and numbers). We should reserve some namespaces 
and numbers for each eStream server. Following is the current list of EMS codes between 
SLiM/Clients. 



#define EMCCNULL 

#define EMCC_ACQUIRE_ACCESS_TOKEN 

fldefine EMCCRENEWACCESSTOKEN 

#define EMCCRELEASEACCESSTOKEN 

#define EMCCREFRESHSERVERSET 

#define EMCCGETLATESTAPPINFO 

#define EMCC GET SUBSCRIPTION LIST 
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• In addition to any data (pages etc.), an EMS calls needs to return a number of codes to 
communicate success/errors. Following structure provides a container for returning 
multiple return codes. By convention, we'll put either EMCR_FAILURE or 
EMCR_SUCCESS in Code[0]. 

typedef struct { 

ulnt32 SizelnBytes; 

uM32 ReturnCodeCount; 

ulnt32 ReturnCodes[]; 
} EMCReturnCodes; 



tfdefine 


EMCR 


SUCCESS 


#define 


EMCR 


FAILURE 


#define 


EMCR 


USER AUTH FAILED 


#define 


EMCR 


ACCESS TOKEN INVALID 


#define 


EMCR 


SUBSCRIPTION JNVALID 


#define 


EMCR 


LICENSE NOT AVAILABLE 


#define 


EMCR. 


LICENSE ALREADY HELD 


#define 


EMCR 


EVICTION NOTICE 


#define 


EMCR 


EVICTION MUST UPGRADE 


#define 


EMCR 


"EVICTION END MEMBERSHIP 


#define 


EMCR 


"eviction NO PAYMENT 



Acquire Access Token 



Caller: eStream Client. 

Callee: SUM Server. 

RPC Code: RPCC_ACQUIRE_ACCESS_TOKEN 

IN ulntl28 Subscription© 

IN String UserName 

IN String Password 

OUT EMCReturnCodes ReturnCodes 

OUT eStreamAccessToken AccessToken 

OUT ulnt32 RenewalFreq 

OUT eStreamServerSet AppServerSet 

OUT eStreamAppInfo LatestAppInfo 



Client will use this interface prior to starting an eStream application to grab the license. It 
accepts a subscription id, which clients received when an app was subscribed, and password; it 
replies with at least a list of return codes and possibly, the access token, its renewal frequency 
and a set of servers that can serve this app. 

AccessToken - Client treats them as opaque data structures and renews them within its renewal 
frequency. 
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RenewalFreq - This ulnt64 is the number of seconds the access token is Valid for once it 
receives it. You probably don't want an absolute count (i.e. # of seconds since epoch) since 
clients can interpret it differently due to clock skew. 

ServerSet - When a client gets an access token, it will be given a list of app servers that can 
serve the particular app. The ServerType member of eStreamServerSet structure will be 
SERVERTYPEAPP. The list is specific to each app and should be managed as such. 
LatestAppInfo - SliM servers will pass information about the latest app version (root) using this 
structure. Refer to client eFS design for more detail This structure will always be passed; client 
will ignore it if it already has the latest version. 

Note: There is a big difference between major and minor upgrades: a major upgrade 
would be going from word 98 to work2000 (where app ids must change) where as a 
minor upgrade (app ids will not change) means applying a patch or a service pack. 
LatestAppInfo tries to transparently propagate latter (minor upgrades) to end users 
without requiring end users to unsubscribe/subscribe apps. Major upgrades will require 
end users to go back to the ASP web server and change subscriptions. ASP can force the 
end user into changing subscriptions (word 98 to word 2000) using 
EMCRJBVICTION JV1UST JJPGRADE error code. 

Return Codes 

Success: EMCR_SUCCESS 

Failure: EMCRFAILURE, plus one of following: 

• EMCR JJSER_AUTH_FAILED - Can't authenticate user with specified passwd. 

• EMCRLICENSENOTA VAIL ABLE - License is not available. 

• EMCR LICENSE ALREADY HELD - If the user is already holding the license, SLiM 
server returns this error code along with the access token that is held & its renewal 
frequency. Most common cause of this error is when an end user tries to run an eStream 
app on two different machines simultaneously. NOTE: returned token doesn't give the 
right to run the application and should be treated as a denial of access token. Reason for 
returning the token/renewal interval is to allow the client software can effectively release 
the token, wait some time (>= renewal frequency) and re-try. 

o The reason client has to wait is because SLiM servers will not communicate the 
list of 'bad' access tokens to the app server. 

• EMCR_EVICTION_NOTICE -ASP wants to stop the user from using ASP resources. 
Server may also add code that describe the reason like 'no payment' etc. Note that no 
access token will be given! This may change in the future to allow some grace period. 

o EMCR_EVICTION JVTUST JJPGRADE - This type of eviction means the ASP 
wants the end user to stop using this particular application in favor of another 
(major) version of it. For example, Word 98 to word 2000. 

Renew Access Token 



Caller: eStream Client 

Callee: SLiM Server 

RPC Code: RPCC_RENEW^ACCESS_TOKEN 

IN String UserName 

IN String Password 
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IN/OUT eStreamAccessToken AccessToken 

OUT EMCReturnCodes ReturnCodes 

OUT eStreamServerSet AppServerSet 

OUT ulnt32 RenewalFreq 



Clients will use this interface to renew the access token before it expires. Client will specify the 
old access token and if there are no errors, get back EMCRJSUCCESS, a new access token, new 
app server set (ServerType field of eStreamServerSet structure will be SERVER TYPE APP) 
and new renewal frequency. Upon getting the new app server set, client must remove the old app 
server set for this application. If for some reason, the access token is expired, SliM server will 
treat this request as 'Acquire Access Token' and may return error codes possibly from that 
interface (this is one of the reason for asking for usernames/password). 

NOTE: Unlike Acquire Access Token, it is not returning LatestAppInfo or 
EMCR_EVICTION_MUST_UPGRADE error codes because once the app is running, we can't 
upgrade apps while running. 

ReturnCodes 

Success: EMCR_SUCCESS 

Failure: EMCR FAILURE, plus one of following: 

• EMCR ACCESS TOKEN INVALID - Access token is invalid. 

• EMCRJBVICTION JsfOTICE -ASP wants to stop the user from using ASP resources. 
Server may also add code that describe the reason like 'no payment' etc. 

o NOTE: will NOT return EMCRJEVICTIONJV1USTJUPGRADE. 

• Error codes from Acquire Access Token. 

Release Access Token 



Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: RPCC_RELEASE_ACCESS_JOKEN 

IN eStreamAccessToken AccessToken 

IN String UserName 

IN String Password 

OUT EMCReturnCodes ReturnCodes 



Client will use this interface to release the license held by the specified user. It should be called 
synchronously when the application exits or crashes. The reason for requiring usemames and 
password is to authenticate the identity of the caller against access token owner. The reason for 
proactively releasing tokens as opposed to just letting them expire is because releasing it allows 
the user to re-acquire it (on the same or different machines) without waiting for it to expire. This 
allows the user to do acquire -> release -> acquire without any wait. 

ReturnCodes 

Success: EMCR_SUCCESS 
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Failure: EMCR_FAILURE, plus one of following: 

• EM CRACCES STOKENINV ALID 

• EMCRUSERAUTHF AILED 



Refresh App Server Set 



Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: RPCC_REFRESH_APP_SERVER_SET 

IN eStreamAccessToken AccessToken 

IN ulnt8 BadQoS 

IN ulnt8 NoService 

OUT EMCReturnCodes RetumCodes 

OUT eStreamServerSet ServerSet 



App Server sets are given to a client when an access token is acquired and are automatically 
refreshed when an access token is renewed. However, the client can always refresh its app 
server sets using this interface. Potential reasons for clients to do this: 

- All servers in the current server set are not responsive - NoService = TRUE 

- Servers are up, but client experiences bad QoS (network delays/timouts). BadQoS = 
TRUE 

RetumCodes 

Success: EMCR_SUCCESS 
Failure: EMCRFAILURE, plus one of following: 
• EMCR_ACCESS_TOKEN_INVALID 



Get Subscription List 

Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: EMCC_GET_SUBSCRIPTION_LIST 

IN String UserName 

IN String Password 

OUT EMCRetumCodes RetumCodes 

OUT ulnt32 NumberOfSubscriptions 

OUT ulntl28[] SubscriptionID[] 

A client can ask for the current list of subscribed applications using this interface. SLiM server 
returns the number of apps subscribed and an array of subscription ids. 

RetumCodes 

Success: EMCR_SUCCESS 
Failure: EMCR_FAILURE, plus one of following: 
• EMCR_USER_AUTH_F AILED 
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Get Latest Application Info 



Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: RPCCJLATEST_APP_INFO 

IN String UserName 

IN String Password 

IN ulnt 1 28 Subscription© 

OUT EMCReturnCodes ReturnCodes 

OUT eStreamAppInfo Upgradelnfo 



Any upgrades pending? This functionality is piggy backed on 'acquire access token' interface, 
but there is some value in providing it as an explicit interface. SliM server will give you the 
latest application information block associated with the specified subscription id; the client can 
decide if it already has the latest root (version) or not. 

ReturnCodes 

Success: EMCR__SUCCESS 

Failure: EMCRJFAILURE, plus one of following: 

• EMCRUSERAUTHFAILED 

• EMCR^SUBSCRIPTION_INVALID 



Component design 

Server Common Services 

Following diagram shows the common portion of all eStream servers. Most of these boxes won't 
be described in this document because they are covered in specific documents. 



Logging 




Caching 


Command 
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Config 
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Encryption 






Thread 
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Compression 
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Load 
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■- 



Various Decisions 

• SLiM server will uses an ODBC interface to communicate with the central database. 
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• From eStream client's perspective, all SLiM servers are equal in functionality. This is 
unlike an application server, which can be segmented to serve specific applications. 

• Each SLiM server will have a unique IP/Port combination. Multiple SLiM servers 
running on the same machine can be distinguished by giving them different port numbers. 

• SLiM servers (and app servers) will not assume any default ports; it will rely on an ASP 
admin to configure the port assignment. With help from OTI, ASP admin will determine 
how many eStream servers need to run on a machine and assign a unique number to each 
eStream server. 

Hardware Failover 

There will be a pool of SLiM servers at an ASP site; from eStream client's prospective, each of 
them is identical. When a client subscribes to an eStream application, it gets a set of SLiM 
servers to communicate with. The clients will keep this list in memory and refer to it when 
calling SLiM server interfaces. If it experiences difficulty communicating with a particular SliM 
server, it will try other servers that are part of the server set. If for some reason the server set is 
lost, or all servers in the set are not responding, a client can always go back to the ASP web 
server and refresh its server set . This gives you a transparent (from end user's prospective) 
hardware fail over path. 

The same approach will also work for app server fail-over scenarios; specific differences are 
that: 

1. SLiM Servers, not ASP web servers, will provide the app server set. 

2. App server list will be refreshed automatically, when access tokens are renewed. This 
allows ASP admins to take out servers from the pool by waiting certain amount of time 
(>= access token renewal frequency) and not cause unnecessary client timeouts. 

3. App server sets are specific to each app; SLiM servers are not. 

Load Balancing 

In eStream 1 .0, we will not require a third party load balancers at an ASP site; we'll do minimal 
things at both ends (clients/server) that should be good enough for small to medium size ASPs. 
We may have to test with selective 3 rd party load balancers to see if we can work with them or 
not; but this is an open issue (listed in the open issues at the end of doc). 

In eStream 1 .0, we'll capitalize on the hardware fail-over mechanism to also aid load balancing. 
Following two actions will perform load balancing: 

• When a client gets server sets (app or SLiM servers), it will distribute its hits randomly 
among the server in the set. In addition, clients will also get new app server sets every 
time they renew access tokens. 

• On server side, the monitor will keep track of each server's response times to process 
client's requests. The data gets sorted from most responsive to least responsive and stored 
into the database. Top 'X' servers from this list will be given to the client when it makes 
an explicit request to refresh its app server set, acquires or renews an access token. 
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Testing design 



Interface Testing 

SLiM server is tightly coupled with three components: client, ODBC/Database and monitor; it is 
fairly difficult to isolate it from all of those components for unit testing. A better approach is to 
exhaustively test the client/SLiM server interfaces, which will in fact also test large numbers of 
interfaces to the other two components. The idea is to crank up a client that will make every 
possible SLiM server request and make sure that SLiM server responds accordingly. 

I think it is good idea to create a simple testing framework (that may evolve with time) that will 
simulate a real client to SLiM and app servers. We can do this by writing a program that includes 
common (client/server) data structures definitions, links in our eStream Message Services (EMS) 
component and invokes various interfaces like 'Acquire Access Token*. From SLiM server's 
prospective, this test program is a working client. 

For each client/server interface (i.e. Acquire access token) write a test case (dummy client) that 
will: 

o Assume that we have created a dummy database that has certain users, passwords and 
subscriptions. 

o Invoke SLiM server with all possible input permutations. This isn't too bad since 

most interfaces have 2 to 4 arguments, 
o In the process, ensure that SLiM server returns all possible return values it can. 

For instance, lets assume that Acquire Access Token has following prototype: 
AET(uIntl28 subID, String UserName, String Password); 

TEST BEGIN: 

Assert (AET(NULL, NULL, NULL) returns 

EMCRFAILURE & EMCRUSERAUTHFAJLED); 

Assert (AET(good_sub_id, goodusername, goodj>assword) returns 
EMCR_SUCCESS, an access token, its renewal freq. Etc.); 

Assert (AET(good_sub_id, goodusername, good ^password) returns 
EMCR FAILURE & EMCRLICENSEALREADYHELD); 

Stress testing plans 

Stress testing in general will be common across all eStream servers. I think it will be a good idea 
to invest in a 3 rd party tool that can simulate real-time load on eStream servers and see its 
responses. Rational has various tools such as Visual Test, Robot and Site Load that are worth 
evaluating. 
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Coverage testing plans 

Lot of these items will apply to all eStream components and probably should be covered in a 
separate eStream test plan document. I am not sure if we should do these things before each 
component is done or wait until they are integrated. I just want to state what may be obvious so 
that it is documented: 

• SLiM server will achieve 85% PFA coverage as measured by Rational PureCov. Tests 
used to measure PFA coverage will be reproducible, either by hand or via an automated 
test suite. 

• SLiM server will resolve all memory corruption and memory leak issues as reported by 
Rational Purify. 

• We should have test cases that will exercise all command line options for SLiM server. 

• SLiM server will be code reviewed by at least two peers. 

Upgrading/Supportability/Deployment design 

This document must have a discussion of how the component addresses any specific issues 
related to upgrading, supporting and deployment of e-stream applications. Some examples 
include: error conditions detected and reported by this component, any special hooks this 
component will provide for monitoring, hints for troubleshooting problems, any special hooks 
for debugging this component. 

Open Issues 

This is a list of issues that need to be further investigated or revisited during implementation. 

1 . How do you produce GUDDs on unix servers? Should app ids, user ids, access token ids 
be guids or we should create them by knowing what numbers are already used? 

2. Resolve big-endian - little-endian issues. Owner: Sameer 

3. Meaning of 'eviction' notice is not conveyed to the end user yet. Owner: Client person - 
Ann. 

4. Encryption impact on SLiM servers. Owners: Amit & Igor. 

5. Global name space & number ranges for different components. Owner: Bhaven 

6. ASCII v/s Unicode strings? Owner: Sameer. 

7. Test with 3 rd party load balancers to see if we work or not. Requirements for deployment 
team: tell us which load balancer to certify against and set them up in our future testing 
lab. Owner: deployment team. 
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eStream Web Server/Database Low Level Design 

Bhaven Avalani 
Modified: 

Functionality 

The eStream solution provides a set of account, user, and subscription manage- 
ment utilities. These utilities are provided as extensions to the ASP's(Application 
Service Provider) web server. 

There are three categories of users for these utilities: End User, Group Adminis- 
trator and ASP Administrator. The roles and the capabilities of each of these us- 
ers are detailed below. 

End user for a system is the user who will actually access eStream application us- 
ing the eStream clients. An end user should be able to: 

• Create Account and User attributes. (Username, Password, etc.) 

• Change Account and User attributes. 

• View all available applications in the eStream system. 

• Subscribe/Manage eStream applications. 

• View Account Status. 

1 . List of applications subscribed. 

2. Status of current subscription. 

3. View/Change Billing information. 

4. View/Change Account information. 

A Group Administrator is an administrator for a group of users. An individual 
user is by definition a group administrator for a single user group. Capabilities of 
a group administrator are: 

• (All of single user capabilities). 

• Add delete users from a group. 

• Manage the active sessions for a group. A group manager should be able 
to release licenses from active sessions, thereby kicking out active users. 

• View the billing information. This will probably need hooks to an external 
billing system. 

An ASP administrator manages the overall application system. Capabilities of an 
ASP administrator are: 

• Manage accounts/users/subscription for all users/groups in the system. 

• Manage the application data for a subscription system. 
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1 . Add new applications to the system. 

2. Modify application information for the system. 

3. Provide the pricing mechanism for the applications(?). 
• Manage the servers in the system. 

1 . Configure a server. 

2. Stop/Start a server. This is accomplished by a message to the 
Monitor server. 

3. Get load information for a server. 

4. Get logging information for a server. 

There are essentially two different types of accounts, which the system will support: Sin- 
gle user account and corporate accounts. 

The following licensing mechanisms will be supported by the system. 

• Fixed Duration License. (Typically monthly license). 

• Fixed Duration Floating License. An example of this is n licenses for k users for a 
fixed duration. 

• Indefinite License. 

Description 

There are several key issues that need to be determined for the Web Server architecture. 
The options available in the market to implement these technologies are listed below. 

Web Server: 

• Apache 

• Netscape Server 

• Microsoft Internet Information Server 
CGI Technology 

• Servlet/JSP 

o Tomcat ( from Apache group) 
o JRun (from Allaire) 

• Active Server Pages (available on NT only) 

• NSAPI ( C level API available for Netscape and Apache). 

• ISAPI ( C level API available for IIS and Apache) 

• CGI (Perl/C etc.). 

Database Connectivity 

• JDBC. 
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• ODBC 

• Native. 

Database 

• SQLServer 

• Oracle 

• Sybase 

• Informix 

• LDAP(??) 

The overall proposed solution for eStream LO Webserver release is: 
Apache + Tomcat(for JSP/Servlet) + JDBC + SQLServer. 

The reasons for choosing this combination for the servers are as follows: 

1 . JSP/Servlet is the only technology which is available for cross-platform and cross 
Webserver support. 

2. We need to decide on a single web server to develop and test against for release 
LO. Apache is chosen to be the one as it is popular on Unix and NT platforms and 
it is freely available. 

3. Tomcat(Apache group's reference implementation for JSP/Servlet specs) is the 
preferred CGI technology as it works well with Apache and all other web servers. 

4. JDBC is preferred for database connectivity as its database neutral and works well 
with Java environment of Servlets. 

5. SQLServer is the preferred database for release 1 .O^This contains the scope for 
testing and deployment for eStream 1.0. 

Since all other servers(App Server, SLM Server and Monitor) are C-H- components, the 
following technology combination will be available for Database Access. 

ODBC + SQLServer. 



The data model for the eStream LO database essentially consists of two high level com- 
ponents. The database deployment architecture is shown below: 



Omnishift Technologies, Inc. 



3 



Company Confidential 



eStream Web Server/Database Low Level Design 




Server Management Component: This component's primary responsibility is to man- 
age the configuration, load and log information for a logical server in the system. The 
clients to this component are all the servers and administration manager. A detailed list of 
interfaces for this component is described in the interfaces section. 

User/Account/Subscription Management: This component is responsible for maintain- 
ing the user account and subscription information for the system. The end user using the 
end user interface performs the updates to this component. Slim Server will access this 
component to validate subscriptions. A detailed list of interfaces for this component is 
described in the interfaces section. 

Group Management: This component is useful for managing groups of users. The group 
administrator can only perform updates to this component. A detailed list of interfaces for 
this component is described in the interfaces section. 

Billing Management: This component's responsibility is to provide interfaces to an ex- 
ternal billing system. A detailed list of interfaces for this component is described in the 
interfaces section. 
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Application Management: This component's responsibility is to provide application 
management interface. This component is accessible for updates only by the ASP admin- 
istrator. A detailed list of interfaces for this component is described in the interfaces sec- 
tion. 

License Manager: This component's responsibility is to manage the licenses. SLiM 
server will check out licenses from the license manager. A detailed list of interfaces for 
this component is described in the interfaces section. 



The architecture for the Web Server extensions implementation is shown below: 



Reques^ 



Dispatch Servlet 



Web 
Server 



Response 



JSP 



JSP 



JSP 



Worker Bean 



Worker Bean 



Data Access 
Bean 



The basic elements of this architecture are as follows: 



1 . Every request into the system goes through a dispatcher servlet. This servlet will 
perform initialization, initial validation of the request and miscellaneous checks 
before dispatching the request to a JSP page. A worker bean will responsible for 
performing the initialization. The processing of the incoming request is performed 
at this stage. The request is then dispatched to an appropriate JSP page. 

2. The JSP page will invoke worker beans to access the dynamic data from the data- 
base via the Data Access Bean and the resultant page is sent back to the user. 
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This architecture is illustrated with the following example. 

1 . User sends in a request to update the username and password information in the 
database. Inputs are username, old password, new password. 

2. The dispatch bean will call the user(worker) bean to: 

a. Validate the user's old password. 

i. The user worker bean will make a request to the data access bean 
to access the password for the user. 

ii. The two passwords are compared and the result is returned. 

b. If the password was valid then, update the new password. 

i. Call the data access bean to update the password in the database. 

c. Else return failure. 

3. Based on the success or failure the dispatcher will dispatch the page request to the 
appropriate JSP page. (eg. error .jsp on failure and user jsp on success). 

4* The page will invoke the appropriate the worker bean (error bean or user bean) to 
obtain the dynamic data and send the response back to the user. 

The salient features of this architecture are: 

1 . Presentation and processing logic is separate. Thus, the customer(ASP) can cus- 
tomize the look and the feel of the pages without impacting the processing logic 
as it is segregated. 

2. The data access bean is separated from the worker beans, which are primarily re- 
sponsible for the business logic. This allows us to change the data access layer (eg 
enabling LDAP access) in the future without impacting the system drastically. 



Data type definitions 



The central data structure for Web Server is the database model. The overall database 
model for user and subscription management is shown below. 
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The important features of this data model are: 

Account: Table holding all the billing and contact information for a user or a group. 

User: An end user in the system. A user can optionally belong to a group. 

Group: A group of users. One of the users in the group is designated as the group admin- 
istrator. Each group has a unique account associated with it. 

Application: This table contains the data about various applications in the supported by 
the ASP. 

License: Each row in this table corresponds to the licensing term for a given subscription. 
This table also maintains the active count of the licenses checked out. 

Subscription: This table contains entries for subscription items. A subscription item con- 
sists of user/group, application and license. 

Usage: This table contains the runtime information for a system. SLiM server updates 
this table with access token usage data. A billing system may interface with this table to 
generate billing data. A reporting system may interface with this table to report on usage 
patterns. 

LicenseUsage: This table is responsible for recording checked out licenses in the system. 

The data model for storing the server related information is shown below: 

PK: Primary Key for the table. 

FK: Foreign Key. Used for relations between tables. 

11,12.. : Index Columns. 



Omnishift Technologies, Inc. 



8 



Company Confidential 



eStream Web Server/Database Low Level Design 




The tables in this model are: 

Server: This table contains entries for each logical server in the system. 

Machine: This table contains entries for each physical server in the system 

Configuration: This table contains configuration entries for a given server. The configu- 
ration entries can be hierarchical in nature. Each configuration has the following format: 

Name Valuel [Value2] [Value3] [ParentConfigld] 

Load: This table maintains the historical and real-time load information for a given logi- 
cal server in the system. 

Log: This table maintains the logs for a logical server in the system. The log messages 
saved here are "major" events in the logical server system. A detailed logs stored in a flat 
file on the physical machine containing the logical servers. 

Global Data Structures: 

struct ServerTuple 
{ 

int serverld, 
int type, 

String serverName 

}; 
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struct Couple 

{ 

String name, 
String value 

>; 

For the Access Token and related data structures, please refer to the SLiM server Low 
Level Design Document. The interfaces below will discuss some of the APFs based on 
the these data structures. 



Interface definitions 

The interfaces exposed by various sub-components are detailed below. 

Server Management Component: 

CreateServer 

int CreateServer (ServerConfig* config) 

Jnput: 

Server Configuration. 
Ou tpu t : 

Unique ID for the server. 
Comments: 

Create a logical server with predefined con- 
figuration. 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

UpdateServerConfig 

Bool UpdateServerConfig(int serverld, String name, String value) 

Jnput: 

Server Id 

Config name and value 
Output: 

Unique ID for the server. 
Comments : 

Create a logical server with predefined con- 
figuration. 
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Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

AddMachine 

Bool AddMachine(String name, String domain, String ip) 
Input: 

Machine name, domain and ip. 
Output: 

Success/Failure 
Comments : 

Create a physical machine entry. 
Errors : 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

SetServerLog 

Bool SetServerLog(int serverld, LogTuple log) 

Inpu t : 

Server Id 

Log tuple (data structure in the Logging 
document . ) 
Output: 

Success/Failure 
Coxmnents : 

Add the log data for a server 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerLog 

LogTuplef] GetServerLog(int serverld, int maxrows = 25) 

Input : 

Server Id 

Maxrows: Maximum number of rows to be re- 
turned. 
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Output: 

Array of Log tuples (data structure in the 
Logging document . ) 
Comments: 

Get the log data for a server 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServers 

Server[] GetServersO 
Inpu t : 
Output: 

Array of Server tuples (data structure de- 
fined above) 
Comments : 

Get all the server information 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

SetServerState 

Bool SetServerState (int serverld, short state) 
Jnput: 

Serverld: Unique id for a server 
State: State information for a server. 
Output: 

Bool True/False for success/failure. 
Comments: 

Update the database with current state in- 
formation for a specified server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 
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GetServerState: Obtain the last known state for a specified server 
short GetServerState (int serverld) 
Inpu t : 

Serverld: Unique id for a server 
Output: 

State: State information for a server. 
Comments : 

Obtain the last known state for a specified 
server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerConfig: Obtain configuration information for a specified server 
ServerConfig* GetServerConfig (int serverld) 
Jnpu t : 

Serverld: Unique id for a server 
Ou tpu t : 

ServerConfig*: State information for a 
server. (ServerConfig data structure is defined 
in the server configuration document) . 
Comments : 

Obtain the last known state for a specified 
server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



SetLoadData: 

void SetLoadData (int serverld, int Load) 
Jnpu t : 

Serverld: Unique id for a server 
Load : Load for the server 
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Output: 
Comments: 

Monitor may call this interface to persis- 
tently store historical load data. It is still 
not clear if SLM and application servers will 
store this directly themselves. 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerLoad: 

void GetServerLoad (int serverld, , int maxrows = 25, int** Load) 
Input : 

Serverld: Unique id for a server 
maxrows: Maximum number of rows to be re- 
turned. Default is 25. 
Output: 

Load: Load for the server 
Comments: 

Obtain server component load information to 
manage load balance . 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



FlushLoadData: 

void FlushLoadData (<tuples> LoadData) 
Inpu t : 

LoadData tuples containing <server id, 
server load> values . 
Output: 

Comments : 

Used to flush aggregated load data to the 
databa 
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Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



User/Account/Subscription Management Component 

CreateUser. This API is used to create user record in the system. Arguments will be 
Usemame, Password. 

Bool CreateUser(String username, String password) 

ValidateUser. This API is used to validate user record in the system. Arguments will be 
Usemame, Password. 

Bool ValidateUser(String usemame, String password) 

CreateAccount. This API is used to create account records in the system. Arguments 
will be billing address, credit card information etc. 

Bool CreateAccount(String usemame, <Account Information>couple[]) 
Input: 

Usemame associated with the account. 

An array of names and values for the account. 
AddSubscription. This API is used by the end users/group administrators to subscribe to 
applications. 

Bool AddSubscription(<Subscription Information>couple[]) 

Input: An array of names and values for the subscription. 



UpdatePassword* Used to change user information. Password, username etc. 

Bool UpdatePassword(String username, String old-password, String new- 
password); 

UpdateAccount Used to update the account information. Billing Address etc. 

Bool UpdateAccount(Couple[]) 

Input: An array of name, value pairs for the fields to be updated. 
UpdateSubscription. Used to add additional time to a subscription. 

Bool UpdateAccount(Couple[]) 
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Input: An array of name, value pairs for the fields to be updated. 

GetUserRecord. Used to get current user configuration. 

Couple[] GetUserRecord (String usemame) 
GetAccountRecord. Used to get current account configuration for a user. 

Couple[] GetAccountRecord(String usemame) 

GetSubscriptionRecords. Used to get to subscription records in a database. End user 
may just want to verify what they are subscribed to. 

Couple[] [] GetSubscriptionRecords(String usemame) 

Output : An array of array of couples containing the subscription informa- 
tion for a given user. 

DeleteUser. Used to delete users who are no longer valid in the system. Typically called 
by the ASP admin. 

Bool DeleteUser(String usemame) 
DeleteAccount. Used to delete un-used accounts. 

Bool DeleteAccount(int accountld) 
DeleteSubscription. Used by the ASP admin to remove subscriptions. 

Bool DeleteSubscription(int subscriptionld) 

Group Management Component 

CreateGroup. This API is responsible for creating group accounts in the database. 
Called by the group admin user. 

Bool CreateGroup(String groupName, String admin, String notes) 
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AddUserToGroup. Adds a user to a group. 

Bool AddUserToGroup(String groupName, String username) 
DeleteUserFromGroup. Removes a user from a group. 

Bool DeleteUserFromGroup(String groupName, String username) 

GetActiveSessions, Gets the active sessions for a group. 

Couple[][] GetActiveSessions(String groupName) 

Output: An array of array of couples containing the following information 
for each active session in the system: 

Username 

Licenseld 

StartTime 

EndTime 

Subscription 

Licensing Component 



CheckoutLicense: Checks out a license. 

int CheckOutLicense(int subscription^, long* pStartTime, long* pStopTime) 
Inputs: 

Subscriptionld: Subscription id of the user. 
Outputs: 

>0 for a successful license. The output is the license usage id. 
StartTime for the license. 
StopTime for the license. 
Comments: 

The system should validate the availability of the license. 

Errors: 

INVALID USER ID 
INVALID SUBSCRIPTION 
LICENSE NOT AVAILABLE 
NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 
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RefreshLicense: Refreshes a license. 

int RefreshLicense(int LicenseUsageld, long* pStartTime, long* pStopTime) 
Inputs: 

LicenseUsageld: License usage id. 
Outputs; 

>0 for a successful license. The output is the license usage id. 
StartTime for the license. 
StopTime for the license. 
Comments: 

The system should validate the availability of the license. 

Errors: 

INVALID USER ID 
INVALID SUBSCRIPTION 
LICENSE NOT AVAILABLE 
NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 
EVICTION 



CheckinLicense: Check in a license 

Bool CheckInLicense(String username, int subscriptionld, int licenseUsageld) 
Inputs: 

Username: user trying to check out the license 

Subscriptionld: Subscription id of the user. 

LicenseUsageld: Usage id for the checked out license. 
Outputs: 

Success/Failure 
Comments: 
Errors: 

INVALID SUBSCRIPTION 
NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 



ValidateLicense: Validate that the user has a license checked out. 

Bool ValidateLicense(String username, int subscriptionld, int licen- 
seUsageld ) 

Inputs: 
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Username: user trying to check out the license 

Subscriptionld: Subscription id of the user. 

LicenseUsageld: Usage id for the checked out license. 
Outputs: 

Yes/No. 
Comments: 
Errors: 

INVALID USER 

INVALID SUBSCRIPTION 

INVALID LICENSE 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



DBAcquireAccessToken 

RPCReturnCodes DBAcquireAccessToken(long Subscriptionld, long* pAccessTokenld, 
string UserName, string Password, long* pStartTime, long* pStopTime, long* Applica- 



tionld) 






IN 


Subscriptionld 


Id of the subscription being used. 


IN/OUT 


pAccessTokenld 


-1 if this is a first time access. 


IN 


UserName 


Username string. 


IN 


PassWord 


Encrypted Password 


OUT 


pStartTime 


Start time for Access Token validity. 


OUT 


pStopTime 


Stop time for Access Token validity. 


IN/OUT 


Applicationld 


Id of the application. -1 Default. 


OUT 


RPCReturnCodes 


RPC Return codes. 


Processing: 







This is fairly complex function. The processing involved in this function call is: 

• If this is the first access (ie *pAccessTokenId = -1) then ValidateUser 

• If the Applicationld is -1 then GetAppId 

• If this is the first access (ie *pAccessTokenId = -1) then CheckoutLicense 

• If this is a renewal request: RefreshLicense 

• If there is a failure and it is due to eviction: GetEvictionReason 

Errors: 



#define RPCRUSERAUTHF AILED 

#define RPCR^CCESSJTOKENJNVALID 

#define RPCR^CCESSJTOKEN_EXPIRED 

#define RPCR_LICENSEJN(OT_AVAILABLE 

#define RPCRLICENSEALREADYHELD 

#define RPCRJEVICTION_NOTICE 
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#define RPCR_EVICTION_MUST_UPGRADE 
#defme RPCREVICTIONENDMEMBERSHDP 
#define RPCR_EVICTION_NO^PAYMENT 



DBReleaseAccessToken 

DBReleaseAccessToken(long AccessTokenld) 

IN AccessTokenld 
Processing: 

• Update the Usage table with the appropriate information. 

• Delete the LicenseUsage record. 
Notes: 

• We need a mechanism to release un-released access tokens. The way to do this 
would be to run a stored procedure at demand and at a predefined intervals to do 
this cleaning up. 

EvictAccessToken 

DBReleaseAccessToken(long AccessTokenld) 
IN AccessTokenld 

Processing: 

• Evicts an access token. 



Billing Component 

AddUsageRecord. Called by the SLM server when it releases an access token. 

Bool AddUsageRecord(String username, int subscription^, date starttime, long 
duration). 

GetUsageRecordsForUseiv Used by external billing system. 

Couple[][] GetUsageRecordsForUser(String username) 
GetUsageRecordsForGroup Used by external billing system. 

Couple[][]GetUsageRecordsForGroup (String groupName) 
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Application Management Component 
AddApplication 

int AddApplication(String appname, String version, String description) 
Inputs: 

Appaname: Application name. 
Appversion: Application version 
Description. Application description. 
Outputs: 

-1 for failure to add the application. 
>0 otherwise. Application ID. 
Comments: 

Returns an app id for a newly added application. 

Errors: 

APPLICATION EXISTS 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetApplicationld 

int GetApplicationId(String appname, int version) 
Inputs: 

Appaname: Application name. 
Appversion: Application version 
Outputs: 

-1 for failure to find the application. 
>0 otherwise. 
Comments: 

Returns an app id for a given application. 

Errors: 

NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 

GetApplicationld 

int GetApplicationId(int Subscription^) 

Inputs: 

Subscription^ 
Outputs: 

0 for failure to find the application. 
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>0 otherwise. 
Comments: 

Returns an app id for a given application. 

Errors: 

NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 



GetSubscribedApplicationlds 

Int[]* GetSubscribedApplicationId(String username) 
Inputs; 

Username: username. 
Outputs: 

Array of application ids subscribed by a user. 
Comments: 

Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetUnsubscribedAppIicationlds 

Int[]* GetUnsubscribedApplicationId(String username) 
Inputs: 

Username: username. 
Outputs: 

Array of application ids not subscribed by a user. 
Comments: 

Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

GetApplicationDetail 

Couple[] GetApplicationDetail(int appid) 
Inputs: 

Application Id. 
Outputs: 
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Array of couple for the app id containing: 

{appname, appversion, description} values. 

Comments: 
Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



Component design 

We will discuss some complex scenarios in this section. 
Subscription 

Single New User 

1 . Create the user. CreateUsen 

a. If a user already exists, return error message and go back to 1 . 

2. Create the account for the user. CreateAecount 

a. Get the contact information from the user. 

b. Prompt to get the billing information. The user may decide to not give the 
billing information at this point. 

Corporate group admin creating an account. 

1 . Create the admin user. CreateUsen 

2. Create the group. CreateGroup 

3. Create the account information for the group. CreateAecount 

c. Get the contact information from the user. 

d. Prompt to get the billing information. 

4. Add users to the group. AddUserToGroup. 

a. This method will automatically create the user if they do not already exist 
in the system. 

b. The list of users is accessible to the Group Admin by querying: 

i. Our database GetUserRecords OR 

ii. Some external database. Eg. LDAP directory. 



Single User subscribing to an application 
1 . Validate the user. ValidateUser 
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2. Prompt to get the billing information if the billing information is not already pre- 
sent. 

3. Get the list of un-subscribed applications. GetUnsubscribedApplications. 

a. GetUnsubscribedApplicationlds. 

b. For each app id returned, get the application details. GetApplicationDe- 
tail 

4. For each additional application user wants to subscribe, call AddSubscription 
SUM server checking out an access token to use an application 

1. Call DBAcquireAccessToken. 



Testing design 

This document must have a discussion of how the component is to be tested. Some sub- 
sections could include: 

Unit testing plans 

The follwowing components will be unit tested: 

ODBC connectivity dll for the SliM server and the Monitor. A simple C++ executable 
will be provided to test the SLiM server and the Monitor interfaces. The C++ executable 
will: 

• establish connection to the database. 

• simulate access token calls. 

• simulate the monitor calls. 

The Web servers' servlets and the JSP pages interact with the database using a set of java 
beans. Each of the bean will have a test interface. A simple JSP will be provided which 
will call all of the test interfaces for the beans. The test interface itself will be responsible 
to call all the interfaces that the bean provides in a predefined calling sequence. 

The servlets will be unit tested using a set of html forms which will invoke the servlets. 

Stress testing plans 

Stress testing will be invoked using some external testing tool which can record HTTP 
traffic and replay the traffic for multiple users. Performix and LoadRunner are two possi- 
ble choices. (There may be additional tools available). 
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eStream Web Server/Database Low Level Design 

Coverage testing plans 

Coverage on the ODBC components will use the same mechanism as the rest of the 
server code. Pure Coverage may be used to achieve this goal. 

Coverage on the Java components is an open issue. We need to investigate the appropri- 
ate tools for doing this testing. 

Cross-component testing plans 

The database is the central point for distribution of the data from Web server to the rest of 
the servers. Thus, creating the database data which can be used by Slim server and the 
App server will be a good source of cross component testing plan. 

Upgrading/Supportability/Deployment design 

We will be finally shipping just the java class files and JSP pages to the customer. It is 
assumed that the customer will have the appropriate web server to support JSP 1 . 1 and 
Servlet 2.2. 

Open Issues 

1 . We have assumed that the JDBC implementation will come from Inet software. 
We may need to change to an alternate JDBC vendor based on pricing, quality 
etc. 

2. Tomcat is decided to be the JSP/Servlet engine. Again, this is a freeware and may 
be replaced by a commercial version (Jrun from Allaire). 

3. The web server will need to talk to the Monitor. The messaging component for 
this communication is not well defined yet. 
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Notes 

The following is roughly what's changed since the last version (0.2): 

□ The functional requirements and use cases have been removed. These will be 
documented in the eStream Requirements Document in future revs. 

□ The entire accounting hierarchy (what is a user and account, how are they 
grouped, at what level does billing take place) is undergoing revision, and has 
been removed from here for this version. 

□ Component descriptions should be more consistent now. 

□ The database of user and subscription information in the client block diagram has 
been removed. See the notes below. 

Known issues 

□ The mechanism for how a pathname on a client machine translates into a globally 
unique FileK) for any eStream server is unclear. This is a major design issue that 
crosses many components on both the client and the servers. 

□ The accounting hierarchy and its impact on this design are missing. 

□ If and how copy-on-write will work for writes to the Z file system is quite 
unknown. 

□ Which server manages user/account/group/subscription data is quite uncertain. 
Representing this by a data cylinder was wrong, and I removed this. However, all 
the interfaces specified below for an "ASP web server" are now just plain wrong, 
and the server team needs to suggest the appropriate changes to the HLD for the 
server topology. 

□ The "Server Data Objects" section at the end of this document needs to be 
rewritten, in terms of interfaces that client and server components supply to 
support these data. 

Introduction 

organization uscu is: 

□ Definitions 

□ Block diagrams for both the client and server portions, showing all major 
components 

□ Each component, generally broken down by 

o purpose 
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o functionality 

o global data managed, if any 

o interfaces for use by other components 



To understand the problem being solved in this design, see the "eStream Requirements 
Document" for information. 

Definitions 

account 

A billing entity consisting of a set of users and subscriptions 
user 

An entity authorized to use an account 
subscription 

An agreement between user and the ASP to use an application under terms of licensing. 
license 

Legal right to use an application at any given time. 
account admin 

A special kind of user who can add/delete other users from an account. 
server admin 

Administrator for all the eStream servers and database. 
AppID 

A unique representation of an application. There is one to one mapping of AppIDs to 
apps. 

Within an application, a unique representation of a particular file. 
access token 

This represents the right to run an eStreamed application. The client must acquire an 
access token before accessing any file (e.g., executing) in an eStreamed app. 
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application 

An application is the set of all files and directories, served by an eStream server, that 
make up a subscribed application. For example, any executable file, DLL, icon file, or 
data file associated with an eStreamed version of FrameMaker is part of this application. 

application installation 

This is the process of locally installing all bits necessary to execute an application via 
eStream. Most files in the application can be read or executed via the eStream file 
system ; some must be installed locally. Some configuration data must also be 
downloaded and processed to allow seamless execution of apps. 

app install block 

This is what needs to be downloaded and installed during application installation . It 
might consist of: 

• all configuration files that must be installed on the client machine 

• all registry spoofing information required to run the app 

• all file spoofing information required to run the app 

• the names of all files and directories that make up the application 

• initial prefetch data 

• initial pages for critical application files 

It's quite possible that this app install block is actually an executable file or a DLL that 
performs all actions to make an application ready to run, rather than simply a block of 
data. 

ASP ID block 

An ASP ID block consists of all the information about the applications available to a 
given user for a given ASP, on a given client machine. Since a user might belong to 
multiple accounts for an ASP, this represents all subscribed applications for all accounts 
for that user. 

Such data might consist of: 



• passworu 

• ASP contact info (IP address, URL, etc.) 

• list of subscribed apps 

o is the app installed on this machine? 
o serial number for app 

o ADRM server(s) to use for validation of this app 
o App server(s) to use to retrieve the app install block 
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o App server(s) to use to retrieve app file data 
. last time stamp when the ASP was checked for new subscriptions 

client certificate 

The client certificate for a client machine is a digital signature used to identify it to 
eStream servers. We anticipate this to be used for requests that don't require an access 
token i.e., a valid license. For example, retrieving the app install block, or data for 
eStream application files that don't require license validation. 

client machine 

This is a computer on which an eStreamed application executes. It may host multiple 
registered users, and a single user can install the eStream client on multiple client 
machines. 

eStream client 

This is the aggregate of all the software required on a client machine to subscribe to, 
install, and execute an eStreamed application. 

eStream File system 

The eStream file system (or EFS) is a distributed file system with prefetch and caching 
functionality. All file data and metadata accessed through the EFS is subject to license 
validation before being available from a server. 

license validation 

The act of validating a license means gaining an access token . Generally, before an 
eStream application can be run on a client machine, the validity of using this application 
by the current user must be checked. This check is done when certain files associated 
with the application are accessed; an eStream server is contacted to perform this check 
and return an access token. 

subscription serial number 

Each application that a user is associated with a serial number. This identifies both the 

or>n 1,V at ; r>n tV, e ncer in">"*1v or,/1 KonrP ran he, checked e?Sl"!y H"f"nff licence 

vauuatjuii . 

Block diagram 

The following are simple block diagrams of the client and server components. Some 
conventions: 
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□ A box represents a logical eStream component. A component may exist as a 
distinct process, or it may be grouped with other components into a common 
process. 

□ A line between components represents an interface call from one to another. If A 
calls B, there's a arrow on the end of the line at B. If A and B call each other, 
there's an arrow on both ends of the line. 

Note that data stores are not represented in these diagrams; if a data store is centrally 
managed, then there is a component that has interfaces to allow access to these data. 
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Component descriptions 
Client components 

The client components are all identified in the block diagram above. Very briefly, some 
points: 

1 . A web browser on the client machine will be used for most user interface 
requests: subscribing to applications, requesting subscription and payment 
information, and so forth. Configuration of the eStream client software will be 
done using a Ul which may be different from a web browser. Some thoughts on 
this are listed below. 

2. The eStream cache manager is the heart of the client software, and is the 
component that actually requests file data from the servers. 

3. The license subscription manager has the task of tracking all valid subscriptions to 
applications from an ASP, and tracking which applications have, or need, a 
license validation to access files. 

4. The app install manager's task is to wait until it's told to install a newly 
subscribed application, and then do so. It also keeps track of what needs to occur 
when uninstallirig an application. 

5. The client network interface simply takes requests from the rest of the client 
components, and forwards them on to the appropriate eStream servers. 

6. The eStream file system (EFS, aka the "Z" file system) is a standard kernel-mode 
network redirector. It presents the normal FS interface to the rest of the NT 
executive, and requests data from the eStream cache manager to satisfy requests 
made of it. 

7. The registry and file spoofers are kernel-mode drivers that monitor registry calls 
and file open requests, respectively. 

8. The No Cluster component is a very simple kernel-mode driver that disables page 
clustering for reads. 

Installation Manager 
Purpose 

Installation of the eStream client software is not different than installing any other client 
software package such as Winzip or Office. The eStream client installation is separate 

fmm the m*t»11*it?on oon^mWirm of eStr^ro subscribed *w1ic«*tio"*. S™ne of tf»*» 
poo^uic pieces mat eotrearn woum need to oe installed are iisceti below. 

1 . Device Drivers 

2. Applications Executables 

3. Application Components 

4. Shared Components 

5. Registry entries 
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6. Shortcuts and Start menus 

7. Help Files 

8. Uninstaller 

Once the pieces of the application to be installed are brought together then an install 
program must be constructed. 

Functionality 

The Installation Manager consists of the following sub-components 
Installshield 

Installshield is the industry standard for building installation sets for Microsoft Windows. 
Installshield will take a set of executables and data files and create a media installation. 
The Installshield environment provides a scripting language that will allow a high degree 
of customization of target installation. The essentials issues for any installation are. 

1 . How much of the application does the user wish to install? 

2. Is the users system capable of running the application? 

3. Where does the user wish to install the application? 

4. Does the user have enough space to install the applications? 

Installshield has a wizard that will set up a project. When the install shield program is 
compiled a media must be specified. The most common media types are floppy, CD 
Rom, and Web media builds. For eStream we may have to ask the clients to reboot the 
machine since we are installing kernel mode components that might need a reboot to take 
effect. 

Install From the Web 

This program is another product that is sold by Installshield that will take a complete 
installation set and create a single executable .exe that can be easily downloaded from a 
web site. 

Uninstaller 

Installshield will provide an uninstaller when it builds the install program. 

■ 0 - . - > • •-• ^ - 

There are three ways that Installshield can patch the system registry. 

1 . Run regsvr32.exe on self-registering .dll files. When the uninstaller is run it will 
use regsvr32.exe /u to un-register the .dll file. 

2. Patch the registry statically. 
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3. Patch the registry based on Installation Options from the install shield script 
program. 

Artwork 

The Installshield program for eStream will require a splash screen and possibly one or 
two other artwork components. 

eStream Client UI Module 

The eStream Client UI module is a client component, currently expected to be running in 
user space. 

Functionality 

The eStream Client UI module supports reporting eStream-specific error & informational 
messages to the client user and soliciting replies when appropriate. It allows the eStream 
client user to view and change the list of applications currently installed on the client 
system and the list of ASP accounts currently known to the client system. 

Interfaces 

ReportMessageToEStreamClientUser(IN message) 

Display specified message in EStream Client UI message window. 

QueryEStreamClientUser(IN message, OUT response) 

Display specified message in EStream Client UI message window and solicit yes/no 
response for return to caller. 

Installed Applications UI 

The Client UI interface allows the user to request that the list of the applications currently 
installed on the client be displayed. The Client UI Module gets this list by calling 
AIM/GetAppInstallListO- 

The Client UI interface allows the user to select an application from this list to be 

unuwtfalled The Client UI Module calls A.TMAJniPRtallAonO to arcnpiplish tVii> 

The Client UI interface allows the user to enter the information necessary to get a new 
application installed. The Client UI Module calls AIM/InstallApp() to accomplish this. 

The Client UI interface allows the user to request that the list of applications currently 
installed on this client be exported to a file, in a form which would allow that list to be 
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imported on another client. The Client UI Module calls AIM/ExportlnstalledAppsO to 
accomplish this. 

The Client UI interface allows the user to request that a list of applications that was 
exported by eStream running on another client be imported from a file & installed. The 
Client UI Module calls AIM/ImportAndInstallApps() to accomplish this. 

Known ASPs UI 

The Client UI interface allows the user to request that the list of ASPs currently known to 
the client be displayed. The Client UI Module gets this list by calling ???. 

The Client UI interface allows the user to select and connect to an ASP in the list. The 
Client UI Module accomplishes this by ???. 

The Client UI interface allows the user to select an ASP from this list to be deleted. The 
Client UI Module calls ??? to accomplish this. 

The Client UI interface allows the user to enter the information necessary to record 
information about a new ASP account. The Client UI Module calls ??? to accomplish 
this. 

The Client UI interface allows the user to request that the list of ASPs currently known to 
this client be exported to a file, in a form which would allow that list to be imported on 
another client. The Client UI Module calls ??? to accomplish this. 

The Client UI interface allows the user to request that a list of ASPs that was exported by 
eStream running on another client be imported from a file & installed. The Client UI 
Module calls ??? to accomplish this. 

eStream Cache Manager 
Purpose 

The eStream Cache Manager (ECM) is a client component, currently expected to be 
running in user space. Its goal is to: 

□ Handle all file requests from the eStream file system, either by using previously 

□ intelligently use preietching oi tile data to reduce latency of pages requested from 
the EFS. 

□ Work with the license subscription manager to insure that all applications have 
appropriately validated licenses before their files are accessed. 
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Functionality 

The ECM handles the volatile & non-volatile eStream cache on the client machine. It 
performs demand fetching and prefetching from the appropriate server(s), using profiling 
data or heuristics. Based on the client's observed behavior, it compiles updated profiling 
data, which may periodically be uploaded to a server. 

Interfaces 

The ECM takes requests from the EFS driver, and makes requests to the client network 
and LSM modules. 

In the descriptions below practically every call could fail for a variety of reasons. The 
associated error handling paths are not shown at this level of the design. 

Open/Create(IN Filename, IN FiteOptions, OUT Handle) 

Called from the EFS. 

This does the following basic tasks: 

□ If the filename and mode options correspond to a FilelD that is already known 
and legal to use (from it's cache), it can just return the handle for this file. 

□ Otherwise, it must ask the LSM for an access token for this file. (This request 
may simply return an access token previously created for other files making up 
the application.) 

□ Launch any prefetching and/or active cache loading activities desired for the 
new app. 

□ Request a file handle from an appropriate app server, via the client network 
component. 

□ Return the handle to the caller 

Clo$e(IN Handle) 
Called from the EFS. 
This basically: 



□ Unloads (or marks as victims) app's cached entries as desired 
Read(IN Handle, IN ReadOffset, IN ReadLength, IN BufferPtr, OUT Byte sRead) 
Called from the EFS. 
This does the following: 
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□ Update the profile data for this file 

□ Check the cache for the requested data 

□ If not there, request the appropriate pages from an app server, along with any page 
prefetches that are needed, and place the retrieved data in the cache 

□ Fill in the buffer and return the number of bytes written to this buffer 

Write(IN Handle, IN Buffer, IN Offset, IN Length, OUT BytesWritten) 
Called from the EFS. 

This will use some copy-on- write scheme. It may be as simple as locking the written 
structures in the cache. 

Global Data 

ActiveAppsData Structure 

Table of data with an entry for each application that is currently active on the client. 
Fields for each entry are listed below. 

• AppPrefix 

• AccessToken 

• ServerName 

• FilesOpen 

• LocalPathName 

FilelD Type 

A globally unique identifier defined for each file associated with an eStream application. 
All EStream-managed files have these identifiers to allow a common & unambiguous 
method of file referencing between clients & servers & to simplify switching the client to 
an alternative server. 

ProfileData Structure 

Exact contents of this data structure will be defined in the low level design phase; at this 
point, assume predecessor/successor pairs w/counts. 

\/nfat>t& fi. Mr>n-\/r>!otfta C.&nhjnq Stnjctures 

Exact contents of these data structures will be defined in the low level design phase. 
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License Subscription Manager (LSM) 
Purpose 

The LSM tracks current subscription information and determines the need for license 
validation. It is informed of subscription changes from the client UI, and is queried by the 
ECM to validate accessibility to different applications, based on the license model for the 
subscription to that application. 

Functionality 

The LSM tracks the users subscriptions to different ASPs; it is part of the client 
component downloaded on a client machine. The LSM starts running when the client 
component starts running, and is remains active until it stops. 

The LSM has a few major tasks: 

1. Keep track of what subscriptions the current user has available from all ASPs 

2. Determine which application a given file is a part of 

3. Acquire an access token to validate a license for file requests that require one 
There are two ways that the LSM updates its list of known subscribed applications: 

1. It may be informed of new subscriptions, or of applications that are unsubscribed, 
by the client UI, as part of a browser plugin in conjunction with an ASPs web site. 

2. It may asynchronously poll an ASPs ADRM servers to get updated lists of 
subscribed apps. 

When the users start running any of the subscribed eStream applications — i.e., when any 
eStream'ed file is opened — the ECM queries the LSM before servicing any requests. The 
LSM checks to see which subscribed application this file belongs to, and, if necessary, 
gets the appropriate access tokens from ADRM servers along with the identities of 
application servers that can be used to run the applications; it uses the client certificate 
obtained when the connection to the ASP was made. At the same time, the LSM can 
decide to cache the access tokens and the identities of the application servers and decide 
to serve them directly from its cache. 

The ECM informs the LSM when files open and close, and determines from this when 

can request for additional access tokens when applications are running and the current 
one is about to expire. 

Global Data 

The global data managed by the LSM includes 



Ornnishift Confidential 



Page 14 



eStream 1.0 High Level Design 



version 0.3 



1 . The ASP ID Blocks which are obtained when the user on the machine establishes 
a connection with an ASP from which the user has subscribed applications. 

2. The access tokens and the identities of the applications servers that are obtained 
from the ADRM servers when the user tries to run the applications. 

Interfaces 

The LSM exposes the following set of APIs to the client UI: 
SubscribeApp(IN ASPId, INAppID, IN Licenselnfo) 

This routine in turn will call the App Install Mgr to install the application on the client 
machine. This will return a Boolean stating success or failure. 

UnsubscribeApp(IN ASPId, INAppID) 

This routine will NOT implicitly uninstall the application. Applications must be explicitly 
uninstalled. This will return a Boolean stating success or failure. 

GetAppList(OUT SubschbedAppList) 

This routine will return a pointer to a list of subscribed applications on the client 
machine. 

The LSM exposes the following set of APIs to the ECM: 
CheckAccess(IN Path, OUT Root) 

The LSM establishes a correlation between the Path and the AppID by querying the App 
Install Mgr. This routine in turn may contact the ADRM server for appropriate access 
tokens. This will return a Boolean stating success or failure. At the same time Root will 
get set to the head of the path that identifies the application so that the file system can use 
the same access token for everything under "Roof. 

Begin App(IN AppID) 

To indicate the start of an application. Note: this may happen implicitly during 
CheckAccessQ. 



To indicate the end of the application. Note: this may happen implicitly during 
CheckAccessQ. 

The LSM makes the following API calls. 
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1 . InstallApp(ASPId, AppID) to the App Install Mgr to install the subscribed 
applications. 

2. GetAppId(Path, &Root) to the App Install Mgr to get the Appld from the Path. 
"Root" is explained above. 

The LSM sends messages to the ADRM server for getting access tokens. When a user 
goes to a new machine and installs the eStream client, the LSM obtains the subscription 
information from this server when the user first establishes a connection with it. 

Application Install Manager (AIM) 
Purpose 

The AIM is the contact point for installation and uninstallation of applications on a client 
machine. It gets the requests from the LSM to install applications when the user 
subscribes to applications, and it gets requests from the Client UI to uninstall 
applications. 

Functionality 

The AIM manages application installs on the client machine. It keeps track of what 
applications have been installed on the client machines, where they have been installed 
and the various components that are part of the installation. It contacts the application 
servers (obtained from the ASP ID block) to get the AppInstallBlock. This may be a data 
block, an application or a dll. The AIM uses the AppInstallBlock to then make the 
appropriate calls to the Registry and File spoofers; to install some files on the local disk; 
to "warm" the cache and to update the start menu and other short cuts as needed. 

Global Data 

The Global Data managed by the AIM includes - 

1 . The AppInstallBlock obtained from the app server that is used to do the 
installation. 

2. The AppID->Path co-relation that is required to check for access privileges. 
Interfaces 

lnstallApp(IN ASPId, IN AppID) 

To install the application using a specific ASP seiner to get the AppInstallBlock. 
UninstallApp(IN AppID) 



Omnishift Confidential 



Page 16 



eStream 1.0 High Level Design 



version 0.3 



To uninstall the application from the client machine. 
GetAppld(IN Path, OUT Root) 

To return the AppID given the Path that is being used to open a file/directory on the 
eStream file system. 

GetApplnstallList(OUT InstalledAppList) 

To get a list of the applications currently installed. 

The AIM makes calls to the registry and the file spoofers using the AddRegSpoofEntry, 
AddFileSpoofEntry, etc. APIs. 

eStream client network component 

This section deals with the components that communicate with the servers. 
Purpose 

The client network component is the common point of connection between the rest of the 
eStream client components and the various eStream servers. Any client module that calls 
an interface of a server does so through the network component. 

This component is basically stupid. It knows the protocols needed for communicating 
with the various servers, and it can encode the requested messages via these protocols, 
but it doesn't try to be smart with regard to failover, or authentication rejection, or other 
error conditions. The network component lets its caller deal with such matters. 

One design assumption here is that data is received from an eStream server only in 
response to a request it has made of this server. In other words, all requests originate 
with the client, never from the server. 

Functionality 

The client network component communicates with the following servers for the types of 
requests listed. 

1 . Validate a user for this ASP and get subscription information 

2. Validate a license for a subscribed app 

App server 

1 . Open a file/directory for a subscribed app 
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2. Various file requests on a previously opened file/directory 
Global Data 
Probably none (?). 
Interfaces 

ValidateUser(IN ADRMServer, IN AspAndUserData, OUT Subscriptionlnfo) 

This interface is called by the LSM; it is used both to validate a user and get updated 
subscription information for a given ASP, 

ValidateLicense(IN ADRMServer, INAPPId, IN ClientCertificate, OUT AccessToken, 
OUT AppServerList) 

This is called by the LSM, to get an access token for an application before its file can be 
accessed. 

AppOpenFile(IN AppServer, IN AccessToken, IN FileDesignator, OUT Handle) 

This is called by the ECM, to for any eStream file. Note that this is also used to retrieve 
an AppInstallBlock, when requested via the AIM. Note; the FileDesignator is still 
undergoing design. 

AppReadFile(IN AppServer, IN AccessToken, IN Handle, IN OUT Buffer, IN Offset, IN 
Length, OUT BytesRead) 

This is called by the ECM. 

UploadAppProfileDataRequest(IN ADRMServer, IN ProfileData, OUT Success) 
It's unclear who calls this! 

eStream File System Driver 

The file system driver interfaces with the operating system's installable file system 
facilities, forwards file system requests that it cannot directly satisfy to the ECM, and 
uses the NT File Cache to optimize repeated accesses to the same data. This component 

manager will be (mostly) OS independent. The file system driver resides in kernel space 
and implements a portion of the entire eStream file system. Other components, such as 
the cache manager, the client network interface, and the app servers, implement the rest 
of the eStream file system. These other components are not necessarily kernel-mode 
resident. 
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Functionality 

The eStream file system driver (EFS) will send most requests from the operating system 
to the cache manager. It will interface with the standard NT File Cache Manager to avoid 
sending redundant requests to the cache manager. And it must support functionality for 
the ECM to notify it when the data structures it has cached have become invalid. 

Global Data 

The only globally- visible data managed by the file system driver are various things that it 
may cache. This includes both file data pages as well as directory contents. These data 
are relevant to the ECM, because it may want to invalidate the contents of the caches if it 
finds a newer version of a data page or finds that (visible) directory contents have 
changed. 

Interfaces 

These are the logical interfaces that are exposed to the ECM. The EFS has standard file 
system interfaces that are used by the NT Executive, but these are not listed here. 

lnvalidatePage(IN FileHandle, IN PageOffset) 

Invalidates the specified page for the specified file handle in the cache. 
lnvalidateDirectory(IN DirHandle) 

Invalidates the specified directory's contents in the cache. This may result in the eStream 
file system driver sending directory change notifications. 

ShutdownFileSystemflN Force) 

Attempts to shut down the file system. If Force is true, the file system will be shut down 
regardless of whether any processes still have handles that are open on the file system. If 
Force is false, this routine will return an error if there are any open file handles. After the 
file system is shut down, any attempt to access the file system will result in errors rather 
than being forwarded on to the cache manager, until StartFileSystem is called. 

StartFileSystemQ 
eStream cache manager. 



Omnishift Confidential 



Page 19 



eStream 1 .0 High Level Design 



version 0.3 



Virtual Memory Clustering Disabling Driver 
Purpose 

The VM clustering disabling driver (aka NoCluster) disables virtual memory clustering 
under Windows. While we don't fully understand all the implications, using this driver 
substantially reduces the average file system paging request size and can dramatically 
improve performance of eStream, especially on slower connections. 

Virtual memory clustering, as implemented in Windows NT/2000, is intended to improve 
performance when paging to and from physical disks. If possible, we would like to 
disable clustering only for those threads/processes that will be doing a significant amount 
of I/O to the eStream file system. 

Functionality 

The VM clustering disabling driver maintains a set of criteria for threads whose 
clustering should be disabled. It will make decisions about which threads should have 
clustering disabled without contacting any other components. 

Global Data 

The only data managed by this component are the criteria for selecting threads whose 
clustering should be disabled. In the simplest implementation, this would be nothing, and 
the driver would disable clustering for all threads. Whatever tables it uses will be 
resident in the kernel, and the driver will be able to access them as needed without 
making calls to a user-mode component to read them. 

Interfaces 

The interfaces for this component are minimal. Clustering may be enabled or disabled, 
and the criteria for which threads to manipulate can be changed. 

StartDisablingClusteringQ 

This interface notifies the driver that it should begin disable virtual memory clustering, 
using the currently specified criteria. 

This interface notifies the driver that it should stop disabling clustering. Note that due to 
implementation problems, we do not support actual unloading of the driver, though it can 
be removed from the system by a reboot. 

ChangeClusteringCriteria(IN Criteria) 
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This interface allows the caller to cause the driver to change the criteria it uses to select 
threads it should manipulate. The criteria might be specified in the interface, or it might 
specify a file that the driver should read for the new criteria. 

QueryClusteringCriteha(OUT Criteria, OUT Active) 

This interface allows the caller to find out what criteria the VM clustering disabling 
driver is currently using, and whether or not it is currently active. 

File Spoofer 
Purpose 

The purpose of the file spoofer is to redirect file system accesses from some non-eStream 
drive. This may be necessary in order to support applications running under eStream that 
are hard-wired to access files in a specific location. The file spoofer may also be used if 
we are interested in providing a version of some system file different from the one 
actually on the client machine. 

Functionality 

The file spoofer will intercept File Create calls for files that we are interested in spoofing 
and ensure that these creates are redirected to a file we specify. The redirection could be 
to a file on the Z file system, or to another, non-eStream'ed file. 

File open is a very common occurrence, so the file spoofer must operate quickly. The file 
spoofer should maintain in-kernel whatever data structures it needs to make a spoofing 
decision. 

Global Data 

The file spoofer must maintain a database indicating which files to spoor, which file to 
replace them with, and possibly which processes should be spoofed. All of this 
information must be kept in-kernel so spoofing decisions can be made quickly. This 
database may change depending on which eStream apps are currently installed or 
running. 

Interfaces 

Causes the file spoofer to begin file spoofing, using the current spoof database. 
StopFileSpoofingQ 

Causes the file spoofer to stop spoofing, but does not change the spoof database. 
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AddFileSpoofEntry(IN SpoofEntry) 

Adds an entry to the spoof database. It is not necessary to stop and restart the spoof 
database to add an entry. 

RemoveFtfeSpoofEntry(IN SpoofEntry) 

Removes an entry from the spoof database. It is not necessary to stop and restart the 
spoof database to remove an entry. 

QueryFileSpoofDatabasefOUT SpoofEntryList) 

Queries the current contents of the spoof database. 

ReplaceFileSpoofDatabase(IN SpoofDB) 

Replaces the entire spoof database. It is not necessary to stop and restart the spoof 
database to perform this action, and it is considered atomic. 

1.4 Registry Spoofer 
Purpose 

The purpose of the registry spoofer is to provide to eStreamed and other applications 
registry entries for eStreamed apps, and to capture registry writes by eStreamed apps so 
they can be purged from the registry or shipped to other clients for configuration 
ubiquity. 

Functionality 

The registry spoofer must redirect registry reads and writes. Because registry accesses 
are quite common, the redirector should be able to service registry spoofs without 
forwarding the requests to a user-level process. 

Global Data 

The registry spoofer maintains a spoof database of which registry entries to spoof, and 
which processes to spoof them for. This database should be kept in-kernel so that the 

snoof decisions can be made ouickly. The spoof database trmv change deoendincr on 

vvii^i vjuoiuii o^piiohuoiii> ukC cUiiCiiuy installed ui rUiiiiiiig. 

Interfaces 

StartRegSpoofingQ 

Causes the registry spoofer to begin spoofing using the current database. 
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StopRegSpoofingQ 

Causes the registry spoofer to stop spoofing. This does not change the registry spoof 
database. 

AddRegSpoofEntry(IN SpoofEntry) 

Adds an entry to the registry spoofing database. It is not necessary to stop spoofing in 
order to do this. 

RernoveRegSpoofEntryflN SpoofEntry) 

Removes an entry from the registry spoofing database. It is not necessary to stop 
spoofing in order to do this. 

ReplaceRegSpoofDatabase(IN SpoofDB) 

Replaces the entire registry spoofing database. It is not necessary to stop spoofing in 
order to do this, and it is considered an atomic operation. 

QueryRegSpoofDatabase(OUTSpoofEntryList) 

Queries the contents of the registry spoof database. 

Server components 

The servers described below are logical servers. Note that a single server machine can 
serve all functions for a small ASP; alternatively, farms of servers can be used to provide 
the functionality of a single logical server. 

NOTE: The distribution of servers and the functionality provided by them are 
somewhat uncertain to date. In particular, exactly who manages the actual accounting, 
user, and group data is undecided. The group producing the LLD for the eStream servers 
need to flesh this out. For now, this document assumes that the ADRM server ultimately 
manages these data, and supplies interfaces to callers to access these data. 

The servers described are: 

] An A dt?m server handles useT/accoi^t/^ibscxmti on i Hptsimwaopniwt pjcwell^s 
vciuuauiig licenses. 

2. An ASP web server is a front-end for requests to add users, subscribe to 
applications, and do various user and application level queries. Generally this 
forwards these requests to an ADRM server. 

3. An application server handles requests to open and read eStream files. 

4. A profile data server will receive uploaded profile data from a client machine to 
enable better initial profile and prefetch maps in eStream sets. 
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ADRM Server 
Purpose 

The Account/Digital Rights Management (ADRM) server is responsible for: 

□ Managing data related to users, the groups they belong to, and the applications 
they are subscribed to 

□ Validating the licenses for applications executing on clients 

□ Tracking all outstanding licenses currently in use 

Functionality 

Client machines sends requests to the ADRM server to add or delete subscriptions, to 
receive an access token to execute an application, and to manage their account/group/user 
relationship. 

Access tokens have an expiration time, so the client must reacquire them at regular 
intervals. When an eStreamed application exits, the client informs the ADRM server to 
release the access token. Any outstanding access token not released or reacquired within 
the expiration time will be automatically released by the server. 

Interfaces 

AcquireAccessToken(IN Userlnformation, INAppId, OUT AccessToken, OUT 
AppServerList) 

This is called by the eStream client to gain validate a license before executing an 
application. 

This is used to insure that a user has the right to use a particular app in a subscription 
from a specific account. The server returns an access token and a list of app servers from 
which the client can access the application file data. If the user doesn't have a valid 
license to use the requested application, a failure message is sent to the client. The server 
writes the start time of this application usage into the database for billing processing. 

RenewAccessTokenflN OldAccessToken, OUT NewAccessToken, OUT AppServerList)) 
This is called by the eStream client. 

The server receives a message from a client to renew its access token before the 
expiration of the token. The server returns a new access token and a list of app servers. 
This allows the server to redirect the client to a different app server in case it knows of 
changes to the list of available servers. Once the token is expired, the ADRM server 
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writes the end time of this usage information into the database and the client must 
reacquire the access token before files for this application are available to it. 

ReleaseAccessToken(IN AccessToken) 

This is called by the eStream client. 

The client returns the token to the server when the eStream app terminates so other 
clients can acquire the token. The server writes the end time of this usage information 
into the database for billing processing. 

AddApplicationServer(IN AppServer, IN ApplicationList) 
This is called by an application server. 

The ADRM server is informed of the availability of a new application server. The 
ADRM server adds this new app server to its list of app servers. 

RemoveApplicationServer(IN AppServer) 

This is called by an application server. 

The ADRM server is told of the removal of an app server. It must remove this app server 
from its list of such servers to prevent any clients from using that server. 

AddApplicationServerApplications(IN AppServer, IN ApplicationList) 
This is called by an application server. 

The ADRM server is informed of the availability of a new application on a given app 
server. The ADRM server adds this new app to the list of applications that the server has 
available. 

RemoveApplicationServerApplications(IN AppServer, IN ApplicationList) 
This is called by an application server. 

The ADRM server is told of the removal of an application for an application server. 



This is called by a server UI tool, or possibly by other ADRM servers. 

The administrator monitoring, reporting, and management tool UI program can query the 
ADRM server for load information. The server logs all client requests to acquire access 
token. This raw information can either be sent directly to the UI program or it can be 
preprocessed before sending to the UI program. 
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GetErrors(IN TimeCriteria, IN ErrorType, OUT ErrorList) 
This is called by a server UL 

The admin UI program can query the ADRM server for any errors. The errors can be 
categorized by type of errors or errors that occur between certain time periods. A small 
sample of the possible ADRM server errors includes: client access token timeout, failure 
to read user information from the account database, failure to get the license information, 
failure to write usage information into the database, etc. . . 

GetlllegalAccesses(IN TimeCriteria, IN AccessType, OUT AccessList) 
This is called by a server UL 

The admin UI program can query the ADRM server for any illegal accesses. The illegal 
accesses can be categorized by type and time period. A small sample of the possible 
ADRM server errors includes: failure attempts to access ADRM server with bad 
password repeatedly in a small time period, failure attempts to use a particular license, 
any access attempts from a non-typical IP address ranges for a particular account, etc... 

GetApplD(IN AppName, OUT AppID) 

This is called by a server UI. 

Returns a unique identifier associated a particular application 
SetApp(ID IN AppName, IN AppID) 
This is called by a server UL 

Stores a unique identifier associated a particular application 

Application Server 
Purpose 

The application server is there to handle read requests for files accessed by eStream 
clients. Any file accessed on a client through the EFS can have this read request passed 

to ar> ar>r> server. 

Functionality 

This will be the hardest working eStream server. It will respond to both synchronous 
(demand fetching) and asynchronous (prefetching) page requests from many different 
clients, for many different types of applications and files within those applications. 
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Interfaces 

GetFilelnfo(IN AccessToken, IN FilelD, OUT Filelnfo) 

This is called from an eStream client. Given any file within an eStream application, 
return metadata about it. The access token is provided for validation. 

ReadFile(IN AccessToken, IN FilelD, IN Length, IN Offset, OUT Buffer, OUT BytesRead) 

This interface is called by an eStream client, and will allow the client to access any 
eStreamed application file and AppInstallBlocks. How the FilelD for an AppInstallBlock 
is achieved is unclear at present. 

OpenFile() / GetFilelDQ 

Note: This is a placeholder for an API that may be needed. This depends a lot on the 
eventual communications between client and server for associating a file pathname with a 
FilelD. 

ASP web server 
Purpose 

This describes, of course, only those interfaces on an ASP web server that relate to 
handling eStreamed applications. 

Logically, the ASP web server is the backend web interface for user requests — e.g., get 
billing information, subscribe to a new app, or request a list of all possible apps a user 
can subscribe to. In the current model, the web server doesn't actually handle these 
requests, but instead passes them on to the appropriate eStream-centric server. 

NOTE: The following interfaces are not updated from the previous version! They 
were written with the assumption that the web server actually manages all the data 
described above. We need the server team to suggest the changes that should take 
place here! 

Functionality 

AddADRMServerQ 

The ASP Web Server is informed of the availability of a new ADRM server. The ASP 
Web Server adds this new ADRM Server to its list of online ADRM Servers. 
Periodically, the ASP Web Server can query the list of ADRM Servers for its load 
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information. When a new client connects to the ASP Web Server, the client can be 
informed of the subset of ADRM Servers with the least load. 

Callers: ASP Web Server 

Input: 

• ADRM Server IP 

• list of Account Ids that the ADRM Server supports. 
Output: 

• success or failure 
RernoveADRMServerO 

The ASP Web Server is told of the removal of an ADRM Server. It must remove the 
ADRM Server from it's locally cached list of ADRM Servers to prevent any future 
clients from using that particular ADRM Server 

Caller(s): ASP Webserver 

Input: 

• ADRM Server 
Output: 

• success or failure 
ValidateSubscribedUserO 
Inputs: 

• SubscriptionToken 
Outputs: 



Validate subscribed user is called by the ADRM server to check out a license. 

• Find account(accountNumber) 

• Find user(usemame) 

• Find license(SubscriptionlD) 
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If a license is found to be available, check it out and return OK else return 
NO_LICENSE AVAILABLE. 

Add/RemoveAccount() 

Input: 

• ownerUsername 

• ownerPasswd 

• billinglnfo 

Output: 

• accountNumber 

Creator must supply info for the first user, who is also the account owner. 

Add/RemoveSubscribableApp() 

Input: 

• application 

• AppServer locations 

• name description 

AddAccountUserQ 
Input: 

• account number 

• user name 

• initial password 

Add/Remove/lncremenWecrementFloatingUcenseQ 
Input: 

• account number 

• subscription 

• number available 

Add/RemovePerilserLicenseQ 
Input: 
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• account number 

• user 

• subscription 
Compile AccountUsageQ 
Input: 

• account number 

Add up and report all the usage by members of the account. 

Clear AccountUsageQ 

FreeLicenseQ 

Input: 

• user 

• subscription 

Frees a license that had been previously checked-out by a user. 

GetUserPhvelegesQ 

Input: 

• account number 

• username 

Check privileges of a logged in user for the purposes of allowing user/subscription 
management and other account changes 

ShowCheckedoutLicensesQ 

GetAccountlnfoQ 

Input: 

» account numoer 

• username 

ListPossibleSubscriptionsQ 
Input: none 
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ListCurrentSubscriptionsQ 
Input: 

• accountNumber 
CreateSubscriptionQ 
Input: 

• account number 

• license data (depends on license type) 

• ApplicationPackage 

Output: 

• subscription© 

• ADRM server names. 

Builder 

Does not talk to any other module. Probably an associated set of tools managed via a 
script plus manual procedures that create intermediate data files, and finally produce the 
eStream set. 

Farm Manager 

Simply takes user commands and input, activating the following functions on the actual 
Application Server process: 

StopServerfboolean graceful) 

ConfigureServer(configjparameters) 

EnableEstreamSet(applD) - informs the app server that the set is ready to be served 
DisableEstreamSet(applD) - opposite of above 

level stuff would be an extension of the above. Synchronizing the availability of apps 
between the Application Server and ADRM/Account DB must also be handled; at least 
initially, a human administrator should be able to flips the final switch. 
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Monitor 

The monitor utility is responsible for monitoring the overall health of the system. It is 
responsible to report server status, server traffic, illegal access etc. It will ping the 
Application Server and the ADRM server to gather the statistics and display them. 

Server data objects 

This section needs work! What should be here? 

eStream Set 

What is an eStream set? It consists of: 

• Page prediction map, indicates likelihood a page will be referenced successively 
after another page. Used to enable accurate prefetching by the client. 

• Spoofer info, stuff to initialize register & file spoofer to enable application to run 
on the client. (ITARD & "Spoofed file mapping list") 

• Application content. This includes all files of the application in possibly multiple 
subtrees (stuff from C:, Z:, Common Files, etc.). These are all then placed under 
a single application root directory which is "mounted" under the eStream file 
system under the application's directory ("Microsoft Office" or whatever). This 
directory structure is then processed to create a special eStream metadata file to 
represent it, and map file names to FilelDs, one assigned to each file for the app, 
used by the client. All of the application files can then be placed into a single 
large file with a FilelD index in front to allow the Application Server to map 
requested FilelDs to offsets in the large application content image file. 

• The above takes care of everything in the AppInstallBlock except for possible 
COM dlls that might be necessary (TBD what mechanism generates these). 



Account/Subscription Database: 

Description: 

The Account Subscription database manages all the data required to manage accounts, 

users and subscription rights, to log usage, and compile billing for application rental. This 
document describes the data model for the db, the list of accessor APIs, and finally, a list 
of scenarios that exercise these APIs. 



Data Model 

The following is a description of the data model for the Account/Subscription Database. 
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There are two types of records described. 

Static Objects are objects that are stored persistently in the Account/Subscription 
database. The attributes of each record are divided into two categories: 

• Owns: specifies data that is actually contained in and managed by the object. 
These are the attributes that make the object what it is. For example: an account is 
not an account unless it has billing information. Each item (except the ASP) must 
have only one owner, but may have many references. 

• References: Attributes that create associations to other first-class objects, making 
navigation from one item to another simpler. 

Transient Objects represent data structures that are created in order to pass information 
from place to another. For the most part, they provide shorthand identifiers for static 
objects. 

Each attribute listed below has the format: 

Type: name - (optional) description 
For attributes that don't yet have a defined type, the type is undefined 

Static Data (Database records) 

ASP - The top-level container. 



• List<Account>: accounts - all of the accounts 

• List<Subscription>: subscriptions - all of the available subscriptions 

• List<Applications>: applications - all of the applications currently served by 
Application Servers 

References: 

• None 

Account - Collection of attributes that make up a single account: 



Owns: 



Owns: 



Number: accountNumber - number that identifies this account 
List<User>: users - all of the account users 
Undefined: billinglnfo - Currently undefined type 
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• List<License>: licenses - all of the licenses (per-user and floating) that are 
managed by the account 

• List<UsageRecord> usage - list of records that describe usage of all account users 

References: 

• User: owner - a user who created the account - must be one of the users 
User - a single user of an Account 



• String: username 

• String: password 

• Undefined: Role - specifies permissions on the account, i.e. owner, administrator 
vs. regular user 

• Undefined: Userlnfo - Real name, contact info etc 

References: 

• List<License>: licenses currently held by user 

• List<License>: licenses - licenses to which the user has access. These might be 
either per-user or floating licenses, or a combination of the two. This list might 
also incorporate a means of specifying a preference - for example if a floating 
license and a per-user license are available, use the per-user 



Application - single application that has been made available on one or more application 
servers. 



• String: name 

• String: description 

• Undefined: AppServer location(s) - A location may be a host name that must be 
resolved by the appserver farm, or may be an TP address. 

• Number: AppID 

• A list of FilelDs for this app. 

References: 

• None 

Subscription - An application or group of applications that have been made available for 
rental by a user. 
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Owns: 

• Number: SubscriptionK) 

• String: name 

• String: description 

References: 

• User: user 

• List<Application>: applications - application(s) associated with this subscription 
ApplicationPackage - application(s) that can be rented 

Owns: 

• Undefined pricing 
References: 

• List<Application> applications 

License - base for other licenses - All licenses support the same APIs - check out, check 
in 

Owns: 

• None 
References: 

• Subscription: subscription - subscription for which this license grants rights 

FloatingLicense - one of a fixed number of licenses distributed to a list of users on a 
first-come first-serve basis. Contains all attributes in License plus: 

Owns: 

• Number: numTotal 

References: 

• List<User> holders - the current holders of this license (length = numlnUse) 

• List<User> allowedUsers - list of users allowed to check out this license 
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PerUserLicense - license tied to a particular user- one desktop (machine) at a 
time. Contains attributes in license plus: 

Owns: 

• Boolean: isInUse 

• Undefined - desktop© 

References: 

• User: allowedUser - the (only) user allowed to pull this license. 
UsageRecord - Describes a billable use of the application 

Owns: 

• Undefined: Start time 

• Undefined: End time 

References: 

• Subscription: subscription 

• User: user 

Transient Data 

AccountNumber - integer that uniquely identifies a single account with an application 
service provider. 

UserName - string that uniquely identifies a single user within an account. To uniquely 
specify a user to an ASP, it is necessary to qualify the UserName with the 
AccountNumber of which he is a member. All users have 

UserVerifier - combination of the AccountNumber, UserName, and UserPassword. 
Uniquely identifies a user within an ASP 

• AccountNumber 

• UserName 



SubscriptionID - Integer that uniquely identifies a subscribable application or collection 
of applications within an ASP. 

SubscriptionToken - describes a user-subscription to the ADRM server. Identifies the 
subscription as well as the user trying to access it. 
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• UserVerifier 

• SubscriptionID 

AppID - A unique numeric representation of each eStreamed application. For example, 
word := 1000, excel := 1001, office := 1002 etc. We should be able to represent software 
packages using AppIDs. 

FilelD - Within an eStreamed app, each app-file gets a unique numeric ID. 
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1.0 Introduction 

This document describes the high level requirements for the eStream 1 .0 product. These 
requirements are given first as lists for the client and server components and then as 
scenarios. ■ 

To facilitate the development of follow-on products, eStream 1 .0 does not include 
attributes that explicitly preclude future support of thin clients or of data ubiquity. 



2.0 Client Requirements 

The following are performance and functional requirements for the client portion of 
eStream 1.0: 



ID 


Description 


Priority 


1.0 


eStream client software operates on Windows 2000, 
Windows NT4, & Windows 98 for x86 clients. 


10 


1.1 


eStream client software collects profile information 
during application execution; this information is 
used to improve client-based prefetching. The 
profile data may be uploaded [unless user disables 
this feature] to a server. 


8 


1.2 


eStream client software supports eStream execution 
of top-selling (as represented by Ziff-Davis suites) 
desktop & laptop applications; these applications 
are listed in section 5.0 below. Other applications 
are supported (opportunistically) as well. 


10 


1.3 


eStream client software is obtained via the web or 
via some distribution media & is installed via some 
industry-standard [e.g., installshield] mechanism; its 
installation requires administrative privileges. 

minimize potential stability/reliability problems. 
eStream client software can be upgraded w/o 
reinstallation and w/o breaking installed apps. 


10 


1.4 


eStream client software operates across the different 
languages supported by Windows. 


8 


1.5 


Applications running under eStream have an ave 


8 







interactive response time within 10% of client 
native for connections at 256K bps or higher. 




1.6 


eStream client software is able to operate with only 
16M of available disk space; this is the minimum 
supported configuration. User is encouraged to use 
cache sized for best performance. 


8 


1.7 


eStream client software supports simultaneous 
execution of multiple eStreamed applications, 
including multiple instances of a single application. 


10 


1.8 


eStream client software is able to unambiguously 
reference a particular license for an application. 


10 


1.9 


Applications being eStreamed function in the same 
way that they would if they were installed locally. 


10 


1.10 


eStream client software tolerates server failure [i.e., 
it continues running any active apps and allowing 
apps to be launched], though possibly with some 
delay, assuming that an alternative server of the 
needed type is accessible. 


10 


1.11 


eStream client software detects and tolerates lost or 
garbled messages. 


10 


1.12 


It is difficult to steal an eStream application's code 
or data from the client. 


10 


1.13 


AVhen an eStream amplication i<; beinp install pH on a 
client, the process detects if the app is already 
installed & requests user confirmation to continue; a 
single version of an application is available to the 
user at a time. Install reboot should be avoided, 
unless needed to minimize potential 
stability/reliability problems. 


10 

IV/. 


1.14 


Upon uninstalling an application, all application- 
specific changes to the client system are removed or 
undone. 


940 


1.15 


eStream client software makes minimal changes to 
the client system when running, avoiding/hiding 
registry, DLL, & non-Z file system changes as 
needed. 


10 


1.16 


eStream client software makes a run/no run license 
decision quickly enough when an eStreamed 

satisfaction issues. 


10 


1.17 


eStream client software is launched/terminated at 
boot/shutdown, at logon/logoff, or on demand. 


10 


1.18 


User is able to set initial size of client cache & to 
increase the size of the cache later without 
significant performance penalty. 


9 


1.19 


eStream client software does not include explicit 


9 
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ASP logon/logoff to run installed apps; 
ASPIDBlock stored on client machine gets 
AccessToken to execute app in seamless manner. 




1.20 


eStream client software facilitates roaming (i.e., 
running one's eStream apps from a different client 
or moving one's client system to a different site). 


10 


1.21 


eStream allows exporting information concerning 
installed apps on one client to one or more files 
which can be copied to another client. eStream 
server info is not invalid/inappropriate when the 
client is moved to a different venue. 


8 


1.22 


eStream will not allow users to run the same 
application from multiple clients simultaneouslyjf 
the license prohibits it 


10 


3.0 


Server Requirements 




The following are performance and functional requirements for the server portion of 
eStream 1.0: 


ID 


Description 


Priority 


1.0 


eStream provides user and account management 
capabilities. 


10 


1.1 


User Account Creation/Deletion supported. 


10 


1.2 


User Account is able to subscribe/unsubscribe to 
Applications. 


10 


1.3 


User is able to view billing and account 
information. 


10 


1.4 


User is able to change password/address/billing 
information online. 


10 


1.5 


User is able to list all available & subscribed 
applications. 


10 


1.6 


User is able to access online help/doc, including an 
FAQ database. 


10 


1.7 


Omnishift provides interfaces to facilitate customer 

ou^^^n t->y LllliU ptf.ii.icS. 


10 


1.8 


User is able to enter/modify data securely. 


10 


1.9 


Both IE 4.0/later and Netscape Navigator 4.0/later 
browsers are supported. 


9 


1.10 


ASP agent [i.e., special administrative user at the 
ASP] is able to access all user information. 


10 


1.11 


ASP agent is able to disable a user. 


10 
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1.12 


ASP agent is able to modify license information for 
a user. 


10 


1.13 


User is able to add additional users to the account. 


10 


1.14 


Web Server is highly scalable. 


10 


1.15 


Servers are able to operate in non-English language. 


9 


1.16 


ASP may operate eStream system with single 
server. 


9 


1.17 


Flexible access/export of billing information is 
supported, to facilitate 3 rd party billing systems. 


10 


1.18 


eStream server software and eStream apps can be 
upgraded w/o impacting installed eStream client 
software. All upgrades are backwards compatible. 


10 


2.0 


The eStream framework [ASLM Server] provides a 
mechanism to validate the usage of application 
components with respect to billing models. 


10 


2 1 


ASLM DRM server is able to validate users to use 
specific applications. 


10 


2.2 


ASLMDRM server records all usage activity down 
to the granularity necessary to support billing 
models. The granularity will be reasonably large. 


10 


2.3 


ASLMDRM is able to release license on explicit 
request or timeout from the client. 


10 


2.4 


ASLMDRM is portable across a wide variety of 
platforms and operating systems, including but not 
limited to: Windows NT4, Windows 2000, Solaris 
UltraSPARC, and Linux. 


10 


2.5 


ASLMDRM servers are fault-tolerant. 


10 


2.6 


ASLMDRM servers are scalable. 


10 


2.7 


ASLMDRM server is able to report Denial of 
Service attempts. 


10 


2.8 


AS LMDRM server reports illegal accesses. 


10 


2.9 


ASLMDRM is able to register its presence/load to 
the Web Server(s). 


9 


3.0 


eStream Framework provides management and 
monitoring tool (EMMT) to manage the servers. 


10 


3.1 


EMMT is able to start/stop servers in the eStream 
framework. 


10 


3.2 


EMMT is able to monitor server activity for all 


10 


3.3 


EMMT is able to configure the servers. 


10 


3.4 


EMMT is able to provide historical reporting. 


9 


3.5 


EMMT is able to display information graphically 
and in spreadsheet format. 


8 


3.6 


EMMT is able to raise alarms on predefined events. 


9 


4.0 


eStream framework provides a mechanism to 
deploy the application via the eStream Builder. 


10 
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5.0 


eStream framework support a variety of licensing 
models. 


10 


5.1 


Floating license model is supported, 
n User - k Licenses 


10 


5.2 


Names User License model. (Special case n=k) 


10 


5.3 


Time based licenses at billing granularity. 


10 


5.4 


High water mark license. 


10 


5.5 


Node locked licenses. 


8 


6.0 


App Server is able to Authenticate client's accesses 
(via AccessTokens) completely locally. 


10 


6.1 


App Server encrypts returned data (via a random 
key chosen by the client); it must be 
computationally infeasible to steal an application's 
code while it is being distributed or to determine 
which annli cation a client is running. 


10 


6.2 


App Server is as stateless as possible to allow client 
to switch to alternative app server w/o significant 
overhead "Stateless" means that there is no server 
context that would be lost if the server went down; 
one classic example of this is that "file open" is 
recorded on the client, not on the server. 


108 




Ann Server i<; ontimized to resnond to reauests with 
minimal server load, thereby maximizing 
<;calabilitv 


10 


6.4 


App Servers may be grouped along with any 
number of other such servers into a farm with 
minimal inter-server interactions (as to maximize 
scalability), with load balancing support. 


9 


6.5 


App Server communicates with clients thru 
firewalls. 


10 


6.6 


App Server communicates with clients efficiently 
(e.g., via persistent HTTP connections). 


10 


6.7 


App Server is able to install new eStream sets w/o 

having to po down 


10 


6.8 


Ann Server iq rnnn<;t able to rim for lonf? neriods 
without crashes (i.e. no resource leaks, and handles 
most/all failure modes for system operations); 24/7 
operation. 


10 


i - - 


EMMT communicate through a database which will 




include but need not be limited to Microsoft 


SQLServer. 
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4,0 Builder Requirements 



The following are performance and functional requirements for the builder portion of 
eStream 1.0: 



ID 


Description 


Priority 


LQ 


The Builder installation monitor runs in the 


10 




background, when an eStream application is 






installed as part of its preparation or building 






capabilities. ! 




LI 


The Builder installation monitor captures all the 


1Q 




updates to the System Registry that take place 
during the install. 




L2 


The Builder installation monitor records all the files 


10 




created in the two kinds of directories: the install 






directory and the common directories. 




13 


The Builder must be able to gather initial set of 


10 




application profile data. This data consists at least 






of the page access pattern for starting and 






immediately shutting down an application 




LA 


The Builder must package the eStream Set into an 


10 




easily manageable packages suitable for ASP 






administrators to download to their servers. 




1A 


The Builder must be able to collect per-user profile 


8 




data from the Profile Server and merge the profile 






data into a combined data usable for updating the 






profile data in the appInstallBlock. 




1.6 


The Builder should be run in an environment where 


1 A 
10 




no other applications are running. 




1.7 


The Builder should provide functionality to create 






mstallation set(s) for each of the clients eStream 1 .0 






ic oniric? to Qiir>nni"t 




L8 


It should be possible to change the appld of the 


10 




eStream set when an ASP wants to "install" the 






eStream set in order to host it. 






It should be possible to create a merged eStream set 


10 




for a suite of applications. 












by the Builder using a stand-alone tester and not 






require the eStream clienW-server programs. 




1.11 


The appInstallBlock should have support for 


10 




indicating upgrades at the support site 




1.12 


In the process of creating an eStream set it should 


10 




be possible for the user to delete file entries and 
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registry entries manually to "trim" the eStream set if 




she so desires assuming the user knows what she is 


doing. 


1.13 


The Builder should be run in a clean machine with 


10 




as few software installed/upgraded as possible. 


1.14 


The Builder should support individual applications 


10 




in a suite even if the installer of the suite doesn't 




allow installation of individual applications. 


U5 


The Builder must be able to create an initial set of 


10 




cache contents for the eStream client and allow the 


initial size to be selectable by the user or 
automatically. 



5.0 Client Use Cases 

5.1 USE CASE: Installation of eStream client code 

• Obtain eStream client code bits. 

• Install z: file system hooks & setup to have z: mounted at appropriate time. 

• Install eStream client code, which services z: file sys requests from local cache or 
from servers & which handles sideband communication w/ servers, and setup to 
activate estream client code at time desired by user (boot, login, on demand). 

• Install NoCluster.sys to disable page fault clustering at system boot. 

5.2 USE CASE: Installation of application 

• Obtain AppID & App Server name for installation from DRM SLM Server. 

• Download AppInstallBlock information. 

• Perform initial installation & setup for app, after checking system for previously 
installed version of app & issuing any appropriate warnings. 

53 USE CASE: Uninstallation of application 

• Remove all registry/DLL/filesys changes associated with app installation. 

• Remove all other data associated with application. 
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5.4 USE CASE: Uninstallation of eStream client code 

• Remove z: file system hooks, eStream client code, & nocluster.sys. 

5.5 USE CASE: Execution of eStream client code 

• Respond to z: file sys requests and detect when new eStream app is referenced. 

• Support Client Ul requests. 

5.6 USE CASE: Execution of application 

• Obtain Access Token & list of App Servers from BRM SLM Server. 

• Contact App Server(s) as desired to obtain file system data. 

• Respond to running application's requests, collect usage data. Cache portions of 
application, file system info, & user preference info. 

• Detect server connection issues (apparent loss of connection or connection response 
below acceptable threshold) & licensing issues; negotiate with APRM SLM Server as 
needed. 



6.0 Server Use Cases 



6.1 USE CASE: Create an Account 

• Customer brings up browser and connects to ASP Web Server 

• Screen display shows "create account", customer selects and enters required account 
info (billing info, owner userid, pword, etc) 

• ASP Web server writes account info to Acct DB using Add Account where a unique 
Account ID is assigned 

• Account ID is returned via web page 



• Customer brings up browser and connects to ASP Web Server 

• Customer enters their userid and pword 

• ASP Web server contacts Acct DB, using AddUser userid and initial password, gets 
Acct info and displays to Customer 
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• Customer selects "add user" and enters required user info (username, address, email 
etc) 

• ASP Web server writes user info to Acct DB updating account info 



63 USE CASE: Modify Account 

(includes disabling an account or user, removing users from accounts, changing pwords 
etc) 

• Customer brings up browser and connects to ASP Web Server 

• Customer enters their userid and pword 

• ASP Web server contacts Acct DB, passes along userid and pword, gets Acct info 
and displays to Customer 

• Customer selects "update info" and enters desired changes 

• ASP Web server writes updated account info to Acct DB 

6.4 USE CASE: AddSubscription 

• Connect to ASP web server 

• Enter account number, username, password 

• Verify that user is account admin using GetUserPermissions 

• Get list of possible subscriptions (using ListPossibleSubscriptions) 

• Get list of current subscriptions for account (using ListCurrentSubscriptions) 

• Display in page - User chooses a subscription and license type 

• Display a screen to allow the user to configure the license. For a floating license, 
allow selection of users, etc. 

• Call CreateSubscription to compose the new subscription for each user and create 
licenses. 



6.5 USE CASE: Building an eStream set: 

• Start w/aon CO-ROM, and a freshly installed OS fnlus latest service nack?Y 

iii^iuij app iiitO diiVe vCuliiu JUSC be a ACguiUjf iiciWOiK Uiivc/ 

• A special system monitor logs all registry changes and file system changes 
during the install. 

• File system changes to C: during install probably need to be spoofed (or have a 
registry entry point to Z: instead), especially newly added directories, so need to 
do the appropriate thing. 
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• From this log and the actual files as installed on the machine, the eStream set 
builder creates the eStream set, which is a small set of related files. 

• Separately, we need to actually set up the app for eStreaming, then run it and 
collect profile data to seed the initial page prediction map. 



The AppServer UI (interface to user to control an Application Server on a particular 
machine) presents the following management functions: 

A. Starting a server: 

• AppServer UI always indicates whether an AppServer process is up and running 
(and alive w/status), and if present prompts for restarting the current server 
process. 

• Otherwise it goes ahead and starts up the AppServer process and reports any 
errors. 

B. Stopping a server: 

• Simple, just stops any running servers, gracefully, perhaps prompting user for 
ungraceful shutdown if not successful. 

C. Install eStream set: 

• Each server is configured with a specific eStream set directory, under which it 
places (in their own individual directories) the actual eStream set contents (a few 
files on the native file system). 

• User indicates to AppServer UI where to find the eStream set package provided 
by Omnishift. AppServer UI authenticates the package, and verifies its integrity, 
and if successful, unpacks and places the constituent files in the server's eStream 
set directory. 

• Note that it is possible for the eStream set directory to live on a file server shared 
by other Application Server machines, so installation may be required only once 
(otherwise it must happen once per machine, and a separate AppFarm Manager 
is responsible for replicating eStream sets across a farm to ensure the farm 
machines are symmetric). 

• How does the server know a new eStream set is available? Each set is assigned a 
VolumelD, and the set contents can be placed under a directory with the same 

lists the valid VoiumelDs, which the Application Server only reads, so an entry 
is added at the end of the install procedure. The AppServer UI then must send 
some kind of message/signal to the Application Server to have it resync with the 
file (and start serving the new app). 

• Also note that eStream set install is doable without bringing down the server (or 
any server in the farm. 
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• Having done this, it will probably be necessary to notify the PRM SLM and 
Account Servers that a new app is available. With some scripts provided by 
omnishift, this sould be done by a human administrator. They need to know the 
VolumeE) of the app that was installed along with the full name, so that the client 
can initiate an app install procedure via the VolumeED (the server can then 
provide the AppInstallBIock which probably has a fixed reserved global FilelD). 

• Questions: What if app is already installed (want to allow reinstall or force 
remove first)? What if app is being upgraded (probably also should be a remove 
and then install)? 



D. Remove eStream set: 

• First we probably have to disable the application on the ©RMSLM/Account 
servers. 

• This probably will require sending some kind of message to the Application 
Server (if running) to stop serving the given eStream set, and then waiting for any 
active connections to expire. 

• Then we can just remove the entry in the AppList file, and delete the file system 
image. 

E. Configure Application Server: 

• The AppServer UI presents various configuration options to the user (stuff like 
logging, port #, threads, etc.) Some may require restarting the Application 
Server to take effect, others may take effect immediately. 

Another activity that occurs, automatically, is the processing of profile data. It is not 
clear what the page prediction map looks like, but clients will periodically send profile 
data to the Application Server, which aggregates it, and must store it persistently, and to 
allow new clients to benefit from improved prediction. There may need to be a special 
module that can take the aggregated profile data to modify the prediction map. 



6.6 USE CASE: Acquire Access Token 



Where are we? 

• Client PC has installed the subscribed app and has received a subscription token, 
| and the name/IP of PRM SLM Server. 

• Customer is accessing an app file and doesn't have an access token for it, yet. (i.e. 
double clicking z:\word.exe). 
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I Players involved: client - cache mgr, DRM SLM Server and indirectly, 
User/Account/Sub/Rights DB. 

What happens: 

• Client contacts the DRM SLM server and gives: subscription token, user/passwd. 

• DRM SLM server looks into the user/acc/sub/right DB to 

o Authenticates user and password; may return: "invalid user" 

o Authenticates subscription token; may return: "invalid subscription token" 

o Look at the Accounts container and see if any licenses are available. If so, 

check it out by creating a new access token and updating the accounts 

container. It may return: "can't get license". 

• Return an access token to the client and a list of app servers. 



6/7 USE CASE: Process File Request - steady state 

Where are we? 

■ Client has installed the app and has a list of app servers, 

J ■ Client is holding a valid access token that it acquired from the DR MSLM server. 

■ Client, while processing an IRP, needs to access portion of file on the app server. 

| Players involved: client - cache mgr, app server. NO DR M SLM server or no 
user/account/dub/rights DB. 

What happens: 

■ Client contacts one of the app servers and gives: access token, App ID, File ID, 
length and file offset. 

■ App server quickly verifies the expiration date on the access token. 

o It must not need to contact the user/account/sub/rights DB to do this. It 
only cares about the time- validity of the token. If token has expired, return 
some kind of an error back to the client. 

■ App server locates the data and sends it back to the client. 

NOTE: we are simplifying this quite a bit when discussing the scenarios because 

| key question is MUST all app servers host all estream apps ? 
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6.8 USE CASE: Renew an Access Token - steady state 



Where are we? 

| » Client acquired an access token from the DRMSLM server. 

■ While running the app, client sees the needs to renew the access token. This may 
happen synchronously when the user touches one of the app files, or by a timer- 
driven client daemon that periodically renews an access tokens before it expires. 

| Players involved: client - license manager, DR MSLM manager, and indirectly, 
user/accounts/sub/right/ DB. 

What happens: 

| ■ Client sends an access token to the DRM SLM server. 

■ Check the time-validity of the access token. 

| Assumption: DRMSLM server assumes that only valid access tokens can be 

renewed. An expired token implies a lack of renewal, which implies releasing the 
| license. DR MSLM server can try to acquire the license, but there is no guarantee 

that it will succeed. 

o If token is expired app, goto Scenario: Acquire Access Token. 
| ■ DRMSLM server accesses the user/account/sub/rights DB to: 

o Generate a new token that will expire some time in the future 

(configurable parameter), 
o Update the account container in user/account/sub/rights DB. 
I eReturn the new access token. 



6.9 USE CASE: Validate user request for access to an 
application server 

Procedure: 

Receive username, password, machineNodelD, subID and appID from the Client 
Query AccountDB for license to access application appID in subscription subID 

Send FailureReason to Client 
Else 

Send accessToken, appServers to Client 
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6.10 USE CASE: Add subscribable application from an 
account 

Interface Required: 
| BRM SLM Server::AddSubscribedApp(accountID, subID) 
Procedure: 

Receive accountID, and subID from the Client 
Check for valid accountID, and subDD on AccountDB 
If (no valid accountID or subID) then 

Send FailureReason to Client 
Else if (subID is not already subscribed under accountID) 

Add Subscription subID to Account accountID in AccountDB 
Send Success to Client 



6.11 USE CASE: Remove subscribable application from an 
account 

Interface Required: 
| ©RMSLM Server: :RemoveSubscribedApp(accountID, subID) 
Procedure: 

Receive accountID, and subDD from the Client 
Check for valid accountID, and subID on AccountDB 
If (exist subID in accountID) then 

Remove Subscription subID from Account accountID in AccountDB 

Send Success to Client 
Endif 

Send FailureReason to Client 



6.12 USE CASE: Monitor/management tools 

Interface Required: 

DRM SLM Server: :GetTrafficHistorvO 

DRM SLM Server: :GetUsageInfo(userHX appID, subID, accountID) 

DRM SLM Server::GetCuirentTrafficQ 

ORM SLM Server: : AddServer(serverH>) 

DRM SLM Server: :RemoveClient(userID, serverDD) 
DRMSLMServer: :GetErrors() 
DRMSLMServer: :DumpErrors(filename) 
ORM SLM Server: :DeleteErrors() 
AppServer: :GetTrafficHistory() 
AppServer: :GetCurrentTraffic() 
AppServer::GetErrors() 
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Procedure: 

©RMSLM Servers keep track of traffic info. The monitor/management tool can query 
the SRM SLM /App Servers anywhere for traffic info. Some examples of traffic data: 

- Traffic history of particular server on number of clients served per unit time 

- Monitor length time a userBD used application appID under subscription subID and 
charged to accountID 

- Monitor current load information on all servers (SRMSLM server and app server) 

- Allow admin manually add/remove some servers from the pool. 

- Allow admin to kick some clients off the server. 

The monitor/management tool can also be used to display a list of errors logged by the 
servers. 

- Monitor errors and be able to categorize by error type 

- Monitor errors occurring between certain time periods 

- Monitor errors reported by a particular server 

- Manage errors to dump the errors to a file 

- Manage errors and delete a subset of errors 

Finally, the monitor/management tool can check for any illegal accesses. 

- Monitor failed attempts to access DRM SLM Server with bad password, especially on 
repeated failed attempts in a short time frame. 

- Monitor any attempts to use a particular license and failed. 

- Monitor access to DRM SLM Server from non-typical IP addresses for a particular 
account. The server is required to save the history of IP addresses of accesses to 

a particular subscription account. 



6.13 USE CASE: Adding a new application server. 

Summary: 

An application server's functionality is to provide applications eStream sets to client 
application. An application server is generally added to the system to provide greater 
scalability and/or to provide additional application support. 

Actors: 

1 A CT> ^ ^rr>'*nic;frpt^r* PacnAr»c«^^ f^r J~>c* filler* ~<-*~- ror A ^ ^ n t-.«/-+;rvr>- 

±. /\^KM6juvi Server(s): 1 ne ADKM 6LM server neeas to be notified of the 
presence of an additional application server and the services it provides. 

Inputs: 

1 . Application server(AS) installer 

2. Application eStream sets. These may be available from one of the following 
location: AS installer, some other AS or Farm Manager Server(some central 
repository) . 
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3. BRMSLM Server location(This input may not be required based on scalability 
solution that we decide on). 
Processing: 

1 . Using the AS installer install the application server. 

a. <Server install use case to be added here? Later> 

2. Copy the Application eStream sets. There are several options here: 

a. Provide the eStream sets as a part of the installer. 

b. Provide a script to ftp to another Application server and copy the eStream 
sets. 

c. Provide a management tool to manage the copying of the eStream sets. 
From the ASP's perspective this is the best solution. A tool which 
provides tracks the application would be useful to manage the load. 

3. Configure the server. The server needs to know the additional application sets that 
it supports(? This may not be required). 

4. Start the server. 

5. Register the server with other DRM SLM servers. The following options apply: 

a. Multi-cast the "new server and services" message to the DRM S LM 
servers. 

b. Register the server to a local object server which in turn notifies the object 
servers across the system. CORBA model supports this. 

c. Using the resonate model (described below), all appservers are essentially 
the same server, ie Address app.foo.com will point to a set of app servers. 
A new server enabled will resonate software will automatically register 
itself with the resonate scheduler. (How do we make the resonate 
scheduler aware of the applications available on the app servers?) 

Outputs: 

1 . The App server is installed and running with a set of applications available on it. 



6.14 USE CASE: Removing an Application Server. 

Summary: 

An ASP administrator may decide to remove an application server from the system for 
various reasons. Removal of server from the system would result in notification to the 
| rest of the DRM SLM servers that it is no longer available for servicing the objects. 

i. rior ttUiiiiilii>UatOi. 

| 2. BRMSLM Servers. 
Inputs: 

1 . Application server running on the machine. 
| 2. APRM SLM Server(s): The ASRMSLM server needs to be notified of the 
presence of an additional application server and the services it provides. 
Processing: 
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1 . Stop the application server. This will result in the Application server informing 
the rest of the ADR MSLM servers that it will no longer take any requests. This in 
turn may result in an application being unavailable for usage. Depending in the 
framework used, this can be done in one of the following ways: 

a. Multi-cast the message to the ADRM SLM servers. 

b. Just stop the server in the CORBA framework. The local ORB server will 
notify the unavailability of the resource to the rest of the framework. 

c. Using the resonate model to scale would imply that you just stop the 
server the resonate agent on the server will notify the resonate scheduler to 
deregister the servers. (However its not clear if you can also deregister the 
objects served by the server.). 

Outputs: 

1 . ADRM SLM servers are notified of the removal of the resource. 



6.15 USE CASE: Add a new ADRMSLM server. 



Summary: 

The ASP provider may decide to add an additional A DRM SLM server to enhance the 
performance of the system. The additional A DRM SLM server added to the system 
should be accessible to the ASP's Web Server so that it can direct the clients to the 
| PRM SLM server. (This may not be required if we deploy the Resonate model of 
scaling). 
Actors: 

1 . The ASP administrator. 

2. The ASP Web server. 

Inputs: 

| 1. ADRMSLM installer 

2. Web Server location(This input may not be required based on scalability solution 
that we decide on). 
Processing: 

| 1 . Using the ADRM SLM installer install the application server. 
<Server install use case to be added here? Later> 

2. Start the server. 

3. Register the server with ASP Web Servers. The following options apply: 
a. Multi-cast the "new server and services" message to the Web servers. 

servers across tne sysiem. COKJbA mooei supports tins, 
c. Using the resonate model(described below), all ADRM SLM are essentially 
the same server, ie Address adfmslm.foo.com will point to a set of 
ADRM SLM servers. A new server enabled will resonate software will 
automatically register i tself with the resonate scheduler. 

Outputs: 

| The ADRM SLM server up and running. 
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7.0 Builder Use Cases 

7.1 USE CASE: Install Monitoring 



» Query builder for CD media and installation executable(s) 

• Monitor various registry and file updated during installation 

» Merge installation data for all applications in a suite 

» Relocate files from C: to Z: directory 

» Create appInstallBlock and package the appInstallBlock with the application files 



7.2 USE CASE: Profiling 



• Query builder for application executablef s) 

» Monitor sequences of file accesses from OS to the file system as profile data 
» Identify the subset of the profile data as the initial cache contents 
» Merge profile data and initial cache contents into the corresponding 
a ppInstallBlock 



8.0 Key Applications for eStream 1.0 

Winstone99: 
Business: 

Corel®WordPerfectSuite8: Quattro®Pro 8, WordPerfect®8, Netscape Navigatoi®4.04 
Lotus® SmartSuite®: Lotus® 1-2-3® 97, Word Pro® 97, Netscape Navigator® 4.04 
Microsoft®Office97: Access97, Excel97, PowerPoint®97, Word97, NetscpNav® 4.04 
High-end: 

Adobe® Photoshop® 4.01, Adobe® Premiere® 4.2, AVS/Express® 3.4, 
Bentley System's MicroStation® SE, PV-Wave® 6.1, Microsoft® FrontPage® 98, 
Microsoft® Visual C++® 5.0, and Sonic Foundry® Sound Forge® 4.0. 
Content Creation Winstone 2000: 

iviacioiiicuiajLyieamweaver/.O, iNetscapeiNavigator 4.6, Sonic Founary souna horge 4.5 

Please note that release of Business Winstone 2000, which was originally slated for 
H BR has now been postponed until the Fall Comdex & will be called Winstone 
jffjsAs soon as the contents of this suite are released, we should move quickly to 
assess our support for its application set. 
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eStream 1.0 Server Scaling Estimate 



Anne Holler * 




* Version L0 



Introduction 

This document presents an estimate of server scaling for the eStream 1.0 product as 
compared with its chief competitor, the Citrix product as deployed by Personable. The 
document presents relevant attributes of the basic application execution model for each of 
the two products, discusses and gauges the impact of the areas in which server scaling 
differs between them, considers the effects of additional attributes of the two products on 
server scaling, & finally summarizes the differences in expected server scaling in terms 
of a number. Please feel free to challenge the assumptions, methodology, & calculations 
herein, now & as we move forward through the design & implementation phases. 

In the process of developing this server scaling estimate, certain assumptions about user, 
system, & program behavior are made, & certain design/implementation goals of the 
eStream 1 .0 product are assumed. This material is listed in separate sections at the end of 
the document for ease of reference. 

This work does not intend to imply that the user experience of the eStream 1.0 & 
Personable/Citrix products is expected to be comparable with respect to the relative 
server scaling point identified. Interactive response differs between the two products; 
first-hand experience with server-based applications running on New Moon & 
Personable/Citrix and client-based applications running on the eStream prototype 
suggests that the former are sluggish on an ongoing basis with respect to activities such 
as selecting from pull-down menus & the latter are as responsive as native wrt such 
activities, with noticeable delays engendered only when heretofore unused portions of the 
application's functionality are exercised. Reliability also differs between the two 
products; smooth fail-over of an active Personable/Citrix application to another server is 
not supported, whereas such fail-over is included in the eStream 1 .0 design. 

This document does not address an area in addition to server scaling that may be of 
competitive interest to ASPs; that area is network bandwidth differences between 
eStream 1.0 & Personable/Citrix. Though it might seem intuitive that sending application 
pages across a network consumes more bandwidth than sending user input & display 
output, aggressive client caching ameliorates the traffic associated with application 
paging whereas the traffic associated with the display outout can be quite substantial, 

uwvOiuiii^ ±k> iiuiv.wwu o uwwa WAN J. icaiimidj ocivcf 06 V^iniA 1 ♦ ±<i idll alii C [ilCi CaxtCl , 

HARW99]. It may be worthwhile to collect and compare bandwidth data wrt the two 
products. 
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Application Execution Models 



The following are basic attributes, with respect to application server scaling, of the 
application execution models of Personable/Citrix and eStream 1 .0: 

Personable/Citrix Client/Server Application Execution Model 
Application executes on server 

On app page faults & app data reads, page loaded from server disk 
Server handles incoming network traffic for all keyboard & mouse events 
Server handles outgoing network traffic for bitmap display update 

eStream 1 .0 Client/Server Application Execution Model 
Application executes on client 

On app page faults & app data reads, page loaded from client cache, excpt miss to server 
Server handles incoming network traffic for client cache misses 
Server handles outgoing network traffic for client cache misses 

Application Server Scaling Comparison 

The impact on server scaling of executing applications on the client, rather than on the 
server, is expected to be large. Processor & main memory overhead are associated with 
executing applications, including the overhead for fielding interrupts such as mouse & 
keyboard events. According to HARW99, processing power for running applications is 
typically the biggest Citrix product bottleneck, and that is reinforced by the variance in 
Citrix scaling numbers reported wrt Personable [Ernie] and wrt another ASP [Amit], for 
which the main differences seem to be application execution overhead. HARW99 
indicates that Citrix scales at 10 to 45 applications per processor, largely due to execution 
overhead (though some overhead is due to processing page faults & accessing application 
data, which is considered in the next paragraph). It is somewhat difficult to understand 
how to model the relative benefit for this difference (in the sense that an infinite number 
of applications can run on a processor that they are not actually using to execute!); it 
seems conservative to assume we get at least as much benefit from this factor as we do 
from reducing cache miss overhead on the server, so let us have this factor double 
whatever benefit we project from the factor considered in the next paragraph. 

The impact on server scaling of processing application page faults & application data 
reads on the client in most cases, rather than on the server, is expected to be measurable. 

that the server overhead to fetch a page from disk, whether the request came from server 
execution or from client request, is comparable. (Although we can construct file server 
technology in which client requests take less time than server requests, let us assume that 
the overhead to encrypt the response consumes that time savings). Also, let us assume 
that the same number of page faults occur on the client & on the server (which server 
partitioning for Personable/Citrix could render untrue.) Hence, the difference in server 
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overhead for application file accesses is estimated to be equivalent in scale to the 
reduction in the number of references, which is derived from the client cache miss rate. 
Assuming a client cache miss rate of 2%, each eStream application server can handle 50 
times as many clients with respect to this attribute of server overhead. Doubling that 
amount due to the factor described in the previous paragraph, we estimate that each 
eStream application server can handle 100 times as many clients as each 
Personable/Citrix application server. 

It is somewhat difficult to know how to compare the network interface overhead 
component of server scaling between eStream 1 .0 and Personable/Citrix. The loading 
constraints associated with executing applications on the server are expected to limit the 
amount of network overhead presented to a Personable/Citrix application server. 
Depending on the kind of application, significant traffic is generated to support client 
displays, but HARW99 identifies network bandwidth overhead [discussed in the 
Introduction section] - not server overhead - as the scaling problem engendered by this 
traffic. Each eStream 1 .0 client generates little server network overhead, but the 
reduction in server load due to client application execution allows more clients to be 
connected to a given server, possibly straining the network-oriented portions of an 
eStream application server. At this point, let us assume that server network interface 
overhead does not materially impact the relative server scaling of the two products. 

SSL encryption can dramatically decrease server scaling; by more than 70%, according to 
Igor Balabine. He indicates that third party SSL accelerators should be employed to 
remove this overhead. 

Additional Server Scaling Considerations 

For eStream, application installation induces load on the application server to deliver to 
the client the contents of the AppInstallBlock, which may contain registry & file spoofing 
information, initial cache & profile data, file system structure information, etc. 
Personable/Citrix does not have a comparable feature. Installation is expected to be an 
infrequent occurrence and Omnishift is expected to suggest/provide mechanisms to 
smooth out (or specially handle) high peak demand at particular points in time, including 
product or application launch/upgrade. Let us assume that this (managed) overhead 
reduces application server scaling by the equivalent of 0.5% cache miss. Adjusting the 
running total by this amount implies that an eStream application server can handle 67 
times as many clients as a Personable/Citrix application server can. 

does not have a counterpart on Personable/Citrix [except for page fault clustering ;-) !]. 
Given that eStream 1 .0 is targeted at fat clients, client prefetching (which is redundant 
wrt a comparably warmed client cache) is expected to be used sparingly. Let us assume 
that prefetching adds the equivalent of an additional 0.5% cache miss rate; the intuition 
here is that prefetching is essentially engendered when cache misses occur (i.e., when we 
are exercising parts of the app we have not exercised before) and that we get unneeded 
pages a measurable percentage of the time. Updating our running total by this amount 



Omnishift Confidential 



Page 3 



means that an eStream application server can handle 50 times as many clients as a 
Personable/Citrix application server can. 

Both eStream & Personable/Citrix products include several logical servers in addition to 
the application server. Both have an ASP Web Server portal to an ASP's account 
services, from which a user can obtain billing information, get a list of available 
applications, subscribe to new applications, etc. For both, the ASP web server interfaces 
in some way with an account database of presumably comparable complexity. Both 
eStream & Personable/Citrix have server functionality involving getting the license to run 
an application, which is expected to cause similar database overhead; in eStream 1 .0, this 
process involves getting an AccessToken from an ADRM server. However, 
Personable/Citrix does not have eStream 1.0's concept of renewing an AccessToken. It 
is expected that the eStream 1.0 design will take special care that renewal does not add 
significant overhead to the eStream 1.0 ADRM server (by specifying nontrivial billing 
granularity & AccessToken renewal frequency, by ensuring renewal is lightweight, 
perhaps by having some explicit mechanism aside from AccessToken renewal for token 
cancellation in the event of client failure/disconnect, etc.). eStream 1.0 has the concept of 
records containing application profile information being uploaded to a server; it is not 
known that Personable/Citrix has any comparable feature (although the product likely has 
much other user information recorded at the server, including preferences, etc). It is 
expected that eStream 1 .0 design will emphasize minimal server impact for handling this 
data. [Profile data may not be a core deliverable for competing with Personable/Citrix.] 
In summary, it is assumed that server scaling for auxiliary servers is comparable between 
eStream 1 .0 & Personable/Citrix. 

Conclusion 

Based on the discussions in the previous sections, eStream Server Scaling expected to be 
significantly higher than that of Personable/Citrix. Considering client execution benefits, 
client caching benefits, application installation overhead, and prefetching overhead, 
eStream server scaling is expected to be approximately 67 times higher than the 
Personable/Citrix competitive product. For some given Personable/Citrix application 
server that can handle 20 clients, an eStream application server can handle 1340. 

Assumptions Underlying Server Scaling Estimate 

User's application usage pattern [what is run, for how long, what features] does not 
change materially depending on whether s/he is using eStream 1 .0 or Personable/Citrix. 

estimates for his network bandwidth evaluation, but it might be interesting to see if our 
in-house use of common applications matches those estimates. 

Overall application server capacity is adequate for both products. Personable/Citrix may 
deny application server access to clients when insufficient overall application server 
capacity is available, whereas eStream 1 .0 may continue to allow clients to access the 



Omnishift Confidential 



Page 4 



servers without some explicit client capacity cutoff point, given the usual small impact of 
each additional eStream client. However, it is possible that an extreme performance 
collapse could occur on eStream, if client load were to grow so large compared with the 
capacity of eStream's application servers that the lack of server response caused an 
escalating number of redundant requests from eStream clients retrials. This situation 
needs to be avoided, in the eStream 1.0 design and/or in its deployment. 

The increased client servicing capacity afforded an eStream 1.0 application server will 
not uncover some insurmountable system bottleneck never encountered with the limited 
client servicing capacity possible on the Personable/Citrix server, including areas such as 
maximum number of threads, processes, sockets, buffer size for socket send or receive, 
socket listen queue length, network buffer cache, maximum number of file handles, etc. 

eStream 1 .0 clients are fat enough to allow adequate client caching to hit or better the 
client cache miss goal of 2%. Thin clients (which are not the intended design target for 
eStream 1.0) or fat clients with inadequately sized client caches would increase server 
overhead for paging requests & network traffic beyond the levels estimated in this 
document. 

Design Goals Supporting Server Scaling Estimate 

Overall client cache miss rate is less than 2%. Reaching the eStream 1 .0 client 
performance goals also reinforces the need for a low client cache miss rate. We have not 
collected data indicating how large a client cache would be needed to hold application 
pages and file metadata associated with typical application usage as represented by the 
Ziff-Davis benchmark runs. I think it would be useful to have such data, since it may 
influence client cache design & effective cache management policies. 

AppInstallBlock server overhead is no more than the equivalent of an extra 0.5% cache 
miss rate. Installation is expected to be relatively rare, & downloaded material is 
expected to be kept at the minimum size necessary to reach our functionality & client 
performance goals. 

Client wasted prefetch overhead is no more than the equivalent of an extra 0.5% cache 
miss rate. With fat clients & large warm caches, prefetching is expected to be kept at a 
minimum for eStream 1 .0; again, data gathering related to this area may be useful. 
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eStream License Manager (LSM) Low Level Design 

Charles T. Booher 



Functionality 

The LSM tracks the users subscriptions to ASP accounts. It is part of the client compo- 
nent and runs entirely in user space. The LSM is part of the client UI program that also 
contains the Cache Manager (ECM), Application Install Manager (ECM) and client use] 
interface components. The LSM determines when an application has a right to run on a 
client machine. This is done through the acquisition and management of access tokens 
The LSM also provides a server list to the ECM (Cache Manager) for each application 
that is subscribed. The LSM has two main tasks. 

Manage and provide lists of servers for the ECM 

Provide access tokens to the ECM and renew them before they expire. 




File System Driver 



Figure 1 LSM data flow 
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eStream License Subscription Manager Low Level Design 

What does the LSM do? 

The LSM manages a set of tables inside of an mdb database. This database can be ac- 
cessed with ADO or ODBC. This database includes the following tables. 

Application Table: This is a list of all the applications subscribed. 

User Table: List of User account information 

Application Server Table: This is a list of application servers for each application sub- 
scribed by the client. 

Slim Server Table: This is list of all Slim Servers where access tokens can be requested 
for renewal by an application. 

Access Token Table: This is a list of all access tokens granted by the SLIM server or in- 
stalled with an application subscription. 

When does the LSM operate? 
An Application is Subscribed 

When an application is subscribed, the Application Table, Application Server Table, and 
Slim table must be updated to reflect the new subscription. An expired token is added to 
the Access token database. The normal token renewal processes with update the token on 
the first installation, but the sleep loop for the token thread is interrupted so that the token 
renewal thread is woken up. The token renewal thread will see the expired token for the 
newly subscribed application and issue a renewal request to the SLIM Server. 

Application subscription tasks: 

1. Update the tables 

2. Wake up the Access token Renewal thread. 

A user Logs on 

When a user logs on to the system the client will get a complete list of Application Identi- 

are any applications that are no longer subscribed or new application the system does not 
know about a Message box will appear asking the user if they would like to update their 
subscription information. The synchronization of application sets from client to client is 
only handled by the client user interface. Another task started at logon time is the Token 
renewal thread. The token renewal thread scans the token list to see if any token is ready 
to be renewed. The token renewal thread is sleeps on a long timer and is only woken up 
if a new access token is added to the token database. When the sleep timer on the thread 
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expires, the thread will scan the list of access tokens to see if any of them have expired or 
are near expiration. 

Logon Tasks: 

1 . Establish a logon with the ASP Web server. Get a list of all applications ID num- 
bers and check if any have been added or deleted. If there are any adds or deletes 
force the user to go to the Client administration UI. 

2. Update the Application Server Table 

3 . Update the Slim Server Table 

4. Start the token renewal thread. 

An application is started 

When an application is started, the ECM will request an access token from the LSM. If 
the token is expired then a Message box will be presented to the user with information 
about the expiration of the application. The user can go from that User Interface to the 
client UI that administrates application subscriptions. . 

An Application is un-subscribed 

When an application is un-subscribed, the corresponding records in the Application Ta- 
ble, Application Server Table, Slim table, and Access token table must be deleted. 
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Data type definitions 

These tables are stored in an mdb database file. This database file can be accessed using 
ODBC or ADO database drivers. 

Application Table 



A master table of all application that are subscribed. 



Name 


Application 
ID 


Application 
name 


RootFileNumber 


ForcedUpgrade 


Message 


Type 


GUID 128 
bit 


String 


128 bits 


8 bits 


String 



This table describes the data in the eStreamAppUpgradelnfo Structure 1 . This table will 
link application names with Application ID numbers. 



Application Server Table 



This table will link Application Servers with subscribed Applications. This data is used 
by the ECM to get file information. 



Name 


Application ID 


Application Server IP 


Type 


GUID 128 bit 


IP Address 32 bit | 



A particular client application can have more that one server. 



Slim Server Table 



This table is a list of all servers where access tokens can be requested. An application 
can have more than one Slim Server. 



Name 


Application ID 


Slim Server IP 


Type 


GUID 128 bit j 


IP Address 32 bit 



1 Software License and Management Server A mil Patel, Bhaven Avalandi, Michael Bechman, Sameer 
Panwar 
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Access Token Table 



Name 


ATID 


UserE) 


AppID 


Expiration Date 


Type 


GUID 


GUID 


GUID 


Date 



Access Tokens are requested when an application is first subscribed. They are renewed 
before they expire. 



User Table 



Name 


User ID 


User Name 


Password Hash 


Type 


GUID 


String 


String 



A users password is never saved. The user password is fed to a one way hash function 
and the hash output is what is saved to the database. 
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Interface definitions 
Function 1 

BOOL GetToken{ 
GUID AppID, 
int size, 

eStreamAccessToken * TokenData) ; 
Input : 

GUID AppID: the Application ID for which a 
token is being requested. 

Int size: the size of the token data struc- 
ture 

eStreamAccessToken * TokenData: the token 
structure . 

Output : 

The function will return True if a valid access 
token has been granted, false otherwise. The 
function will handle client UI for expired to- 
kens . 

Comments : 

The eStreamAccess Token is defined in the slim 
server documentation. 

Errors : 

Application ID not found. 
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DWORD GetAppServer (GUID appID, int servernum) 
Input : 

GUID AppID, this is the application ID for 
which a server is being requested. 

Int servernum, this number is usually 0, if 
a particular server is not available then 
the ECM will ask for the next server. If 
the Server table does not have a server then 
this function will return -1 

Output : 

32 bit IP address 
Comments : 

A return code of -1 indicates that we have run 
out of servers 

Errors : 

Application ID not found. 
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Component design 
Database access 

Access to the database tables is through either an ODBC or ADO interface. The AIM 
can access the databases directly though these interfaces. The program code that updates 
these tables is considered part of the AIM, but the databases themselves are considered 
part of the LSM. 

Token Renewal Thread 

The token renewal thread is a worker thread that is part of the Client executable. This 
thread is started with AfxBeginThread. 

UINT MyThreadProc( LPVOID pParam ) 
{ 

CTokenRenewalObject* pObject = (CTokenRenewalObject *)pParam; 

if (pObject == NULL || 

!pObject->lsKindOf(RUNTIME_CLASS(CTokenRenewalObject))) 
return 1 ; // if pObject is not valid 

// do something with 'pObject 1 

While(1) 
{ 

if(CheckForTenmination()) 
return 0; 

pbbject->AccessTokenTable->MoveFirst() 

while(!pObject->AccessTokenTable->lsEOF()) 

{ 

if (NearOrAtExpiration(pObject->AccessTokenTabIe->Expiredate)) 

1 

pObject->RenewToken(pObject->AccessTokenTable->TokenlD); 

} 

pObject->AccessTokenTable->MoveNext(); 

} 
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ComputeSleepTime(); 
Sleep(); 

} 



return 0; // thread completed successfully 

} 

// inside a different function in the program 



pNewObject = new CMyObject; 
AfxBeginThread(MyThreadProc f pNewObject); 



Logon Initialization 

This block of code is part of the Client executable when the client first starts up. 
void ClientStartup(void) 

{ 

LogontoASPWeb ( ) ; 
GetSlimServerTable () ; 
GetAppServerTable ( ) ; 

StartTokenRenewalThreadO ; 

} 



Testing design 

The LSM is part of the Client Executable. This test plan will become part of the client 
executable test plan. 
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Unit testing plans 

1 . Initial Application Subscription Install. Does the AIM update the LSM tables 
correctly? This can be checked using Access. 

2. Contact the ASP web server and Update the Application Server Table. 

3. Contact the ASP web server and get a list of applications. 

4. Does the Client UI respond when the list of applications is changed? 

5. Force tokens to expire using Access and see if they are renewed by the Slim 
Server 

6. Force access tokens to expire and prevent the Slim server from granting them. 



Stress testing plans 

Stress happens when Application Server and Slim Servers go down. Stress testing can be 
accomplished by running subscribed applications and then shutting down servers. 

Coverage testing plans 

Every application that eStream is capable of serving will need to be tested. 

Cross-component testing plans 

The LSM communicates with 

ECM 

AIM 

Webserver 
Slim Server 
Client UI Components. 

Upgrading/Supportability/Deployment design 

This component is part of the client executable program. When the client executable 



Open Issues 

Every time the user logs on to the client a list of currently subscribed, applications must 
be downloaded from the server to insure client synchronization. Right now we don't 
have this function defined on the server side. 
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The following proposal is based on the C++ coding standards document available at 
http . //www .possibi lity.com/Cpp/CppCodingStandard .html . This document will 
concisely present the coding standards from the coding standard document. The reader 
should refer to the original document (linked above) for a detailed explanation of the 
standards. This document has the following sections: 

NAMES: This section contains the naming schemes. 

ERRORS AND ERROR CODES: This section contains the error formats and the error 
codes to be used by eStream 1 .0 client and server components. 
FORMATTING: Code layout and formatting guidelines. 
COMMENTS: Guidelines for applying comments to the code. 
LOOSE END: Loose ends. 

Table of Contents: 



NAMES 

ERRORS AND ERROR CODES: 



FORMATTING 
COMMENTS.... 
LOOSE ENDS:. 



2 
4 
4 
6 
7 
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NAMES 



ID 


RULE 


Variables 


• Use upper case as word separators, lowercase for the rest of the word. 

• No underbars 

• First letter of the variable is uppercase unless it is prepended with some 
other letters(see below). 

Examples: 
short Status; 

unsigned long TimeOfDay; 


Pointers 


• Prepend the variable with p. 
Examples: 
string* pName; 
char** ppValue; 


Class Names 


• Use upper case letters as word separators, lower case for the rest of a word 

• First character in a name is upper case 

• For externally exposed components, use the first 3 words to denote the 
component. 

• No underbars ('_*) 
Examples: 

• class ConfigurationManager 

• class Config 


Library Class 
Names 


• Prefix the classname with OT 
Examples: 

• class OJHttpListener 

• class OTServer 


Class Method 
Names 


Same rule as for class names except for interfaces where the rule is: 

• Prefix the interface with the component's name. 
Examples: 

• ECMGetFileld 

• MonitorGetServerSet 


Class Attribute 
Names 


• Attribute names should be prepended with the character 'm'. 

• After the 'm f use the same rules as for class names. 

• 'm' fllwavQ nreredes other name modifiers like Y* ? for nointer 

Examples: 

int miLen; 

string* mpValue; 


Global 
Variables 


• Global variables should be prepended with "g". 

Examples: 

int gFlag; 

Logger gLog; 

Logger* gpLog; 
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Global 
Constants 


• Global constants should be all caps with separators. 
Examples: 

const int A_GLOBAL_CONSTANT= 5; 


Type Names 


• When possible for types based on native types make a typedef. 

• Typedef names should use the same naming policy as for a class with the 
word Type appended. 

Examples: 

typedef uintl 6 ModuleType; 
typedef uint32 SystemType; 


#defines and 
Macros 


• Put #defines and macros in all upper using separators 
Examples: 

#define MAX(a.b) blah 
#define IS_ERR(err) blah 


Function 
Names 


• Use upper case letters as word separators, lower case for the rest of a word 

• First character in a name is upper case 

• No underbars ('_') 
Examples: 

• int SomeBloodyFunctionO 


Enum Names 


• all upper using 9 J separators 

Examples: 

enum PinStateType 

{ 

PINJDFF, 
PIN_ON 

} 


File Names 


• File should be all lower case 

• File name format should be <component>_<sub-component>.* 
Examples: 

monitor_heartbeat.cpp 
core configmanager.c . 
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ERRORS AND ERROR CODES 



The following pound defines should be used for returning all successes and failures. 

#define SUCCESSWITHINFO -1 
#define SUCCESS 0 

#define FAILURE >0 (The number representing an Error ID). 

All error messages will be prepended with an error code. The format for error code will 
be as follows: 

[ERRORID] [Severity] [Error Message] 
where, 

1. ERRORID are unique across the system. 

2. Severity can be one of the following: 

a. 1-Low : A warning which can be ignored. 

b. 2-Medium: A warning which needs to be looked into. 

c. 3-High: Recoverable error in the component. 

d. 4-Critical: Irrecoverable error. Needs admin assistance. 

3. The error message in itself should have the following format- 
[COMPONENT]:[ERROR MESSAGE]: [WORK AROUND] 

Error Ids distribution for client and server are as follows: 

0- 1 000 Server Internal Error Codes. 
1001 - 8000 Server Error Codes. 
8001 - 9000 Client Internal Error Codes. 
9001 -16000 Client Error Codes. 

FORMATTING 

The following formatting policies should be followed by all code. 
Braces Policy. 



if(0=a) 
{ 

} 

else 

{ 
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} 



Indentation/Tabs/Space Policy 

Use the standard Visual C++ settings which are(using Tools->Options->Tabs menu): 

Indent Size: 4 

Auto Indent: Smart 

100 previous lines used for context. 

White spaces should be spaces and NOT tabs. 

VC++: Select "Insert Spaces" option. (This is NOT the default). 

Emacs: Refer http://www.delorie.com/gnu/docs/emac^/emacs_205.html 

Line Size etc. 

1 . Line size should not exceed 78 characters. 

2. There should be one statement per line. The following piece of code violates this 
principle. 

if(a>b)a++; 

Method/Functions Formats. 

1. Methods should preferably be less 50 lines of code. 

2. Methods should not have more than 4 levels of nesting. 

3. Methods should preferably be re-entrant. Non-reentrant methods should be clearly 
marked as such. 

4. Each method/function should be preceded with a comment describing the method: 
/*************************************************************** 

FUNCTION: 

INPUTS: 

OUTPUTS: 

DESCRIPTION: 

ERRORS: 

*********************************************** 
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COMMENTS 



1 . Every decision should have comments. The following keyword are associated with 
decisions: 

a. if, else 

b. while, continue 

c. switch, case, default, break 

d. goto 

e. return 

2. Every class should have comment header with the following format: 

/** ******************************** ******************************** 

CLASS NAME: 
DESCRIPTION: 
FRIEND CLASSES: 
INCLUDES: 
LIBRARIES: 

********************************************** *********************/ 

3. Every function/method should have a header, (described above). 

4. Every file should have a header describing the contents of the file. 

5. Every directory should have a README describing the contents of the directory. 

6. Make GOTCHAS explicit. Use the following format for gotchas. 

:TODO: topic 

Means there's more to do here, don't forget. 
:BUG: (bugidj topic 

means there's a Known bug here, explain it and optionally give a bug ID. 
rKLUDGE: 

When you've done something ugly say so and explain how you would do it 
differently next time if you had more time. 
:TRICKY: 

Tells somebody that the following code is very tricky so don't go changing it 
without thinking. 
:WARNING: 
Beware of something. 
iCOMPILER: 

Sometimes you need to work around a compiler problem. Document it. The 
: ATTRIBUTE: value 

The general form of an attribute embedded in a comment. You can make up your 
own attributes and they'll be extracted. 
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LOOSE ENDS 



The following section notes some loose ends which do not fall in any of the categories 
above: 

1 . Always initialize all variables every time* 

2. Use header file guards against multiple inclusions of the header file. The guards 
would look like: 

#ifiidef ClassNameh 
#define ClassNameh 

#endif // ClassNameJi 

3. Object constructors should just initialize data. (They cannot return errors). Explicit 
Initialize() calls should be made to do any involved work. 

4. Use continue and goto sparingly. 

5. Be "const" correct. Use "const" wherever and whenever applicable. 

6. All classes must have a Default Constructor and a Copy Constructor 
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QUESTIONS TO ASK AN ASP 



ASP Overall 



1 . What is the current solution deployed by the ASP? Why? 

2. What applications are most interesting to the ASP? What platforms do these 
applications run on? 

3. What applications do they currently support? What are the licensing models for 
these applications? * 

4. How often are the applications upgraded? * 

5. What are the bottlenecks faced with the current solution? 

6. How is user data handled? Is there an issue with the handling of user data? 

7. What billing systems does the ASP use? * 

8. What pricing models does the ASP employ? What pricing models are important to 
the ASP for the future?* 

9. What is the growth estimate in terms of the number of users? 

10. Who are the ASP's key partners? 

1 1 . Who are the ASP's key suppliers (or dependencies)? 

12. Who are the ASP's key competitors? 

13. How does the ASP differentiate its products? 

14. Does the ASP have a QOS solution? 

15. What are the ASP's major security concerns? 

1 6. What is limiting the ASP's growth? 

1 7. What are the ASP's current bandwidth requirements? 

1 8. How significant an operating cost is this expense? 

1 9. How does the ASP expect this bandwidth requirement to change over time? 
ASP Infrastructure 



1 . What are the platforms on which the ASP applications run? 

2. What are the platforms on which the ASP servers run? * 

3. Does the ASP maintain the infrastructure? If not, where is it outsourced and why? 

4. What are the current data center space needs? 

5. How significant an operating cost is this expense? 

6. How does the ASP expect this space need to. change over time? 

7. What kinds of load balancing systems does the ASP deploy? * 

8. What are the app-servers (middleware) used by the ASP? 

9. What ports do the servers use in the current solution? * 

" ^ - * ' * m J — uOs-C) wU^ii OCi Vwi uuv^autuiy SUppVJiii 

1 1 . What are the most important features (wish list) for the app-server infrastructure? 

12. Is the ASP willing to add h/w accelerators for encryption? 

13. How much unplanned down time has the ASP been experiencing? 

1 4. What have been the main causes of these down times? 



ASP end user 



1 . Is it ok to ask the user to do reboots? * 

2. How big of a cache can eStream assume? * 

3. Does eStream need to support local installations of the same application? * 

4. What is the minimum performance tolerated by the end user? 

5. What is the typical bandwidth experienced by the end user? 

6. Is the end user always connected? 

7. Is there a need to allow for offline access of applications? 

8. Can eStream assume that the end user has the ability to install drivers? 

9. Is the end user always behind firewalls? 

10. What is the accepted level of security for the client-server communication? * 

11. What is the accepted level of piracy protection needed on the client side? Does the 
ASP need its own Intertrust kind of solution? * 

12. What is the acceptable delay for installing an application? (Basically, what is the 
maximum size of the AppInstallBlock acceptable to the ASP?) * 

13. Does the ASP need an ability to terminate end users in the middle of a billing 
cycle? * 

14. Can the end user run the same application on multiple clients at the same time? * 

15. If the license expires, can the client continue running if the application is in the 
cache? * 

16. Is collecting profile data from the end user acceptable? Is the profile data of use to 
the ASP? 



Deployment 

1 . What makes for a "good" deployment? 

2. Describe a "good" deployment? 

3. How would the ASP like to work with a solution provider to deploy their solution 
in the ASP environment? (i.e. solution provider installs for ASP; help the ASP 
install; ASP installs alone, other) 

4. How would the ASP look to deploy a solution like eStream? (i.e. phased 
implementation; full-blown roll-out, other) 

5. How would the ASP want to receive the eStream solution software? (i.e. 
physically, electronically) 

6. What things would be of concern to the ASP in deploying eStream in the ASP 
environment? 

7. What sort of technical questions would the ASP want answered before 
considering deploying a solution like eStream? 

and administration? 

9. Would there be a need for a Certification Program for eStream Administrators? 

10. What sort of tools/abilities would the ASP require to manage and monitor a 
solution like eStream? 

1 1 . What kind of monitoring solutions does the ASP IT team use? 



Support 



1 . What would the ASP like to see/require in an SLA? 

2. What makes for a "good" support experience? 

3. Who, that the ASP is currently working with, provides "good" support and what 
happens? 

4. What is missing in the current SLA's? 

5. What would the ASP like to see/require in support provided by a solution 
provider? 

6. What is missing from current solution provider support offerings? 

7. To what extent would the ASP work with a solution provider in remotely 
troubleshooting problems? 

8. Would allowing us remote access into a "Monitor" tool/DB or Log running on the 
ASP system to effectively troubleshoot a problem be acceptable? How would the 
ASP see this working? 

9. Would the ASP provide first line support to the end users? 

10. What does the end user support offering consist of? 

1 1 . What would the ASP require from us in order to be able to offer end-user 
support? 

12. Does the ASP actually provide that support or does it have a 3 rd party do that on 
its behalf? 

13. How would the ASP like to be notified of upgrades and patches? 

14. How would the ASP like to receive upgrades and patches? 

Citrix 

1 . What does the ASP like about MS Terminal Services and Citrix offerings? 

2. What does the ASP NOT like about MS Terminal Services and Citrix offerings? 

3. What 3 rd party tools does the ASP use in addition to or in place of Citrix's 
offerings and why? 

4. How well does the Citrix solution scale? Perform? 

5. How would the ASP rate Citrix's support offerings? 

6. If the ASP were redesigning Citrix Support what would be done differently? 

7. How did the ASP deploy the Citrix solution? 

8. What sort of issues did the ASP run into in deploying the Citrix solution? 
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eStream Server Component Framework 
Low Level Design 



Michael Beckmann 




Functionality 

The Server Component Framework provides a common basis on which server compo- 
nents are implemented. The framework provides a number of services such as common 
server initialization and configuration, messaging, state management, logging, and error 
handling. The component framework ties together many of the core utilities provided for 
the server components. 

The advantage of the framework is that heterogeneous server components can be man- 
aged in a consistent manner with the expectation that all server components will commu- 
nicate and behave consistently within the system. 

All server components with the exception of the web server will be built on top the 
Server Component Framework. To make use of the Server Component Framework, a 
specialized server component will need to extend the framework by implementing the 
methods high-lighted in gray. Implementing these interfaces makes the specialized server 
component "plug-able'' within the framework. 




Database Connectivity Layer 
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The following table give a brief description of each of the routines that need to be spe- 
cialized by each server component to make it plug-able into the Server Framework: 



StartProcessing 


Specialized server component routine to request the server compo- 
nent to start processing work. 


StopProcessing 


Specialized routine to request the server component stop processing 
work and transition into an idle state 


UpdateConfig 


Specialized routine to dynamically update configurations while a 
component is either in the processing or idle state. 


HandleError 


Specialized routine to handle the occurrence of an error 



Server State Manager: 

At the heart of the server component framework is the Server State Manager. The server 
state manager is a set of interfaces that initiate and manage state changes within a server 
component. All Server components, by virtue of being built on top of the component 
framework, can be managed uniformly across a deployment. 

The Server State Manager implements a simple state machine that is shared between 
components. It manages the state transitions within the server component. Additionally, 
the state manager maintains current state information for each server component and logs 
state transition history in the event that a server component terminates unexpectedly. 

As specified above, each server component is required to implement a number of 
transition methods, with pre-defined signatures, which the state manager will execute 
when making a state transition. 

The following diagram shows the state diagram and the associated transitions: 
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Message Service: 

The Server Component Framework depends on a message service which is used by the 
Server State Manager and Configuration Manager to communicate with the System 
Monitor. 

The Server State Manager uses the messaging service to listen for state change requests 
from the System Monitor which it satisfies by returning the current state, any up-to-date 
status, and load information. 

The Configuration Manager uses the message service to request configuration informa- 
tion from the System Monitor, Although each server component could easily go to the 
database for configuration information, it has been decided to go through the monitor as 
to save db licensing costs. 

See below for more details on messaging protocols for the Server State Manager and the 
Configuration Manager. Also, refer to the low-level design document for details on the 
design of the eStream Messaging Service (EMS). 



The configuration management utility is used by all server components to manage the 
server configurations. It provides the following functionality: 

• Configuration for a server consists of a set of name - value tuples where the val- 
ues themselves can be a set of name-value tuple. 

• Servers can load the complete configuration from the database (indirectly). 
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• Servers can load the configuration for a given name. 

• Servers can load the configuration from a flat file also. 

On the Server Manger interface, configuration will appear as a table containing name - 
value tuples. The table maybe hierarchical to represent nested structures containing the 
values which can themselves be name values. An example of a simple name-value pair 
would be: 

port 8080 

An example of nested name values would be: 

Applications: 

word, exe windows2000sp3 
excel exe win98sp4 

On a flat file the configurations will always be name-value pairs. To represent one level 
nested structure the format would be: 

Applications word.exe windows2000sp3 
Applications excel, exe win98sp4 

A common set of configurable parameters is defined for all server components. These 
configurations are maintained by the Server Component Framework in collaboration with 
the Configuration Manager. All configuration information is persistently stored within 
the database. The common configurations are used to initialize the server component 
after the component process has been launched. Refer to the configuration table below 
for more details on common configurations. Specialized server components can support 
additional configurations (non-common) depending on the server type. These 
configurations are read from the database and updated when a server component starts 
processing. They can also be updated dynamically while a server component is 
processing through the use of the UpdateConfig interface. 



The list of common configurations include: 



Information 


Supports 
Dynamic 
Config 


State 


Description 








identity is unique witnm a deployment, i his 
ServerlD is not known to eStream clients. Its purpose 
is as a handle to uniquely identify server components. 


ServerType 


No 




Identifies the type of server component. One of the 
following applies: 
■ Primary Monitor 
» Backup Monitor 
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■ Application Server 

■ SLiM Server 


DbUser 


XT 

No 




User name stnng required for database connectivity 
for this server ID 


DbPasswd 


No 




Database password associated with the DbUser 


Dsn 


No 




Data l^ource Name used to access the database. 


PortNum 


No 




PortNumber used for light-weight messaging listener 


Machine© 


No 




Machine ID is used to get at important machine in- 
formation needed for all server components such as: 

■ IP address for the machine server component is 
hosted on 

■ Domain name for the machine 

■ Machines name 


AutoReStart 


Yes 


Any 
State 


Flag indicating that server component process can be 
restarted automatically without manual intervention 


TimeOut 


Yes 


Any 
State 


Specifies the timeout period for the listener. If the 
timeout period is reached. The component assumes 
that it has lost the connection. All Server components 
have a listener by which they receive instructions 
from the primary system monitor. Even the monitor 
has a listener that communicates with the Server 
Admin UI. 



Command Line Utilities: 



The Command Line Utilities component provides a consistent way to define and process 
command line arguments. To use this utility, the using component must define a table of 
arguments, which defines the valid set of arguments, whether or not they are required, 
and any default values. 

Arguments are specified on the command line as name/value pairs. The utility imple- 
ments the following command line syntax to support the name/value pairs. The argument 
syntax is defined as follows: 

<name>=<value> 



name 



value 



Name is an alpha-numeric identifier. The Name can be of arbitrary length as 
supported by the system however shorter names are recommended. Names are 

Any alpha-numeric value. Punctuation characters may also be used. Values are 
case sensitive 



There can be no spaces between the <name>, "=", and the <value> elements. The exis- 
tence of one or more spaces or tabs delineates separation between arguments on the 
command line. 
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Example: server.exe sid=267 dns=oracle user=michaelb passwd=mypasswd 

□ If a named argument is specified more than once on the command line, subse- 
quent arguments will cause a diagnostic to be issued and the argument will be ig- 
nored. 

□ This utility allows the user to specify default values for arguments. If a default 
value is defined then the argument will be processed with its default in the event 
that the argument is not specified on the command line. 

□ This utility allows the user to tag specific arguments as required. If the required 
argument is not specified on the command line this utility will raise a diagnostic 
for the required argument. Not specifying a required argument will cause a fatal 
error. 



The following options are supported: 



sid 


Server Component Identifier. Each server component within a deploy- 
ment is uniquely identified via the sid. The sid is a handle into the data- 
base for accessing information unique to a specific server component. 


dsn 


Data Source Name. A data source name is necessary to establish an 
ODBC connection. Data Source Names are generated by an ODBC ad- 
ministrative tool 


dbuser 


User name. For database access security, all components need to connect 
as a specific user. 


dbpasswd 


password associate with the dbuser 



Logging Utilities: 



All servers and clients in eStream 1.0 need to log the error and access data. Logging en- 
ables component debugging and auditing support. 

EStream Framework should provide logging with the following features: 

• Each component will have an error and optioanally an access log file. The names 
of these files would be <component>_error.log and <component>_access.log. 

• The files will be located in the <eStreaml .0 Root Dir>\logs directory. 

• The error log files will have messages with the following priorities: 

o 4-Low : A warning which can be ignored. 

o 3-Medium: A warning which needs to be looked into. 



o 1 -Critical: Fatal Error. Needs admin assistance. 
• Logging level should be configurable. The following levels are to be supported, 
o 0: Only errors will be logged. This will be the default level, 
o 1 : Errors and Warnings to be logged. 

o 2: Errors, Warnings and Debugging information to be logged, 
o 3 : Errors, Warnings and advanced Debugging (like memory dumps, tcp 
stack dumps etc) to be logged. 
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• Log Wrapping to be supported. The log files will wrap at a predefined size. On 
wrapping the following actions will occur: 

o Any existing <logfile>.bak will be deleted from the system, 
o The current <logfile> will be backed to <logfile>.bak 
o The component will continue logging to the <logfile>. 

For each eStream client and server component logging the log files (component_error.log 
and component_access.log) should be written in eStream 1 .ORootUogs directory. The 
formats for the log files will be as follows: 

Error Log: 

[HEADER] 

[TimeStamp] [Thread ID] [Priority] [Message] 
[FOOTER] 

An example of this log format would be: 



Omnishift eStream Application Server 
Server Started. 

StartTin^^|j^^^^^fcl6:31 :19 -0700 

IP Address^^^^^^^^S 

Logging Level : 3 

it********************************************* 

[14/Aug/2000:16:31:19 -0700] 0 2 -High Cannot connect to the database. 
Invalid Us ername/ Pas sword. 

[14/Aug/2000:16:31:19 -0700] 1 1-Critical Cannot start the HTTP listener 
at port 80. 

[l4/Aug/2000:16:31:19 -0700] 0 1-Critical Shutting down the server. 

Omnishift eStream Application Server 
Server Stopped. 

St0pTime:^^^^|^J^16:35:19 -0700 

IP Address^^^^^^T 

Logging Level : 3 



Format of Access Log Message: 
[HEADER] 

[TimeStamp] [Thread ID] [Message] 
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[FOOTER] 

Data type definitions 

Server State: 



The server components can be in any one of the following states: 



State 


Description 


STOPPED 


If a server is in the STOPPED state then the component 
process is not running. 


IDLE 


Server component is up and running. The server has been 
initialized with the common configuration and the messag- 
ing system has been enabled. 

The listener is actively waiting on the System Monitor for 
transition requests. 

The server component is not processing any work specific to 
this servers specialization. 


PROCESSING 


Server component is actively taking requests and processing 
work specific to its specialization, ie. serving access tokens, 
and application file requests. 


ERROR 


An error has occurred in the system. Unless the server 
component is configured with AutoReStart and ERROR 
state must be manually cleared by the server-side adminis- 
trator. 



Server State Transitions: 



Changes in server component state are initiated either by the System Monitor or directly 
by the server-side administrator for the system monitor. The exception to this is when an 
error condition is raised by a server component. In this case, the component will initiate 
the state change itself. The following state transitions are supported: 



Action 


Description 1 


STARTSERVER 


Server is expected to be in the STOPPED state. 
If a server component is configured to support AutoReStart 
then the ERROR state is also a valid state from which to initi- 
ate this action. 




from any state. 


START_PROCESSflNfG 


Causes the server to change from the IDLE state to the 
PROCESSING state. 


STOP PROCESSING 


Causes the server to change from processing to IDLE state. 


UPDATE_CONFIG 


Request that the server read its configuration from the configu- 
ration manager and change its configuration. 
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RAISEERROR 


Request that the server go to ERROR state. This causes an er- 
ror handler to be called. If the error is fatal it will cause im- 
mediate termination of the server process. 


Finite State Table: 



FSMTableEntry ServerStateMgr::FSMTable[] = 
{ 

{ START, { {ST ART_SERVER,STOPPED, START_SERVER,NULL} , 

{START_PROCESSING, STOPPED ,START_PROCESSING, NULL}, 
{NULLREQUEST, NULLSTATE, NULLREQUEST, NULL}} }, 



{STOPPED, { {STARTSERVER, IDLE, NULL REQUEST, &StartServer}, 

{START_PR0CESS1NG, IDLE, START_PROCESSING, &StartServer}, 
{RAISE ERROR, ERROR, NULL REQUEST, &HandleError}, 
{NULL REQUEST, NULL STATE, NULL REQUEST, NULL}} }, 

{IDLE,{{START_PROCESSING, PROCESSED, NULL REQUEST, 
&StartProcessing} , 

{STOP_SERVER, STOPPED, NULL REQUEST, &StopServer}, 
{RAISE_ERROR, ERROR, NULL REQUEST, &HandleError} , 
{UPDATE CONFIG, IDLE, NULL REQUEST, &UpdateConfig}, 
{NULL REQUEST, NULL STATE, NULL REQUEST, NULL} } }, 

{PROCESSING, {{STOP_PROCESSING, IDLE, NULL REQUEST, 

&StopProcessing} , 

{UPDATE_CONFIG,PROCESSING, NULLREQUEST, 

&UpdateConfig}, 

{STOP_SERVER, IDLE, STOP_SERVER, &StopProcessing}, 
{RAISE_ERROR, ERROR, NULL REQUEST, &HandIeErTor} , 
{NULL REQUEST, NULL STATE, NULL REQUEST, NULL}} }, 

{ ERROR, {{STOP_SERVER, STOPPED,NULL_REQUEST, NULL}, 

{NULL REQUEST, NULL STATE, NULL REQUEST, NULL}} } 

{NULLSTATE, { {NULLREQUEST, NULLSTATE, NULLREQUEST, 
NULL} } } 

h . 

itAOAdgiiig oci *i\-e jri otocui: 

A light-weight messaging protocol is needed to facilitate communication between server 
components. The primary purpose of the messaging protocol is to communicate transi- 
tion requests to the server components. In response, server components communicate 
state, status, and load information back to the System Monitor. 
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The messaging protocol supports two primary message types. 1) Requests for the System 
Monitor to perform on other servers. 2) Requests to the server components themselves. 
These message types are distinguished through the protocol as described below. If the 
receiver ID and the target ED are identical then the request is for the receiver. If the target 
is different than the receiver, the message is for the System Monitor to enact a request on 
the target component. 



All requests are required to be acknowledged. Without an acknowledgement the message 
is considered un-received. 



OpCode | senderlD [ receiverlD | targetID | Data 



The following table describes the protocol used by the Server State Manager in its com- 
munication with the System Monitor. 



OpCode 


Description 


Data 


0x01 


Request for current state 


None 


0x02 


Acknowledgment 


■ Current state 

■ Load info 

■ Status info 


0x03 


Stop Server request. Acknowledged with 0x02 
message 


None 


0x04 


Start Server request. Only valid for System 
Monitor. Acknowledged with 0x02 


None 


0x05 


Start Processing Request. Acknowledged with 
0x02 


None 


0x06 


Stop Processing Request. Acknowledged with 
0x02 


None 


0x07 


Update Configuration Request. This is a re- 
quest for a server component to request its spe- 
cialized configuration information from the 
System Monitor and update itself. Acknowl- 
edged with 0x02. 


None 


The messaging protocol used by the configuration manager is described below: 


OpCode 


Description 


Data 


0x01 


Request a complete reload of Configuration 






items. 


being requested. 
■ array of names of 
configuration items. 


0x03 


Acknowledgement of configuration reload re- 
quest. 


■ Number of tuples 
being returned 

■ Flat representation 
of configuration tu- 
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pies 



Interface definitions 

Server State Manager: 



class ServerStateMgr 

{ 

private: 

ServerState CurrentState; 

static FSMTableEntry FSMTable[]; 



public: 



ServerStateMgr(void); 
~ServerStateMgr(void); 

ServerState SetState(ServerState); 

ServerState GetState(void); 

ServerState ProcessRequest(ServerRequest); 



SetState 



Description: Sets the current state of the server component. 

1 . Log the state change request 

2. Update the state field within the server component in memory data struc- 
tures. 

3. Send message to requester informing them of the successful state change. 
Note: SetState does not update the database directly as in the orginal design. 
The database is updated by the System Monitor once it has received an ac- 
knowledgement. A state transition is not complete until SetState returns suc- 
cessfully and the Monitor has update the database. 

Input: state value to set current state to. 

Output: current state after the new value has been set. If an error occurs will 

go to error state. 

Errors: 

1 . Invalid state argument 

2. Failure to either connect or commit state change to the database. 



| GetState Description: returns the current state. This function does not read from the ) 

-* ' - - . 1 £ _ 

nent is up and running and that it maintains a valid state. 
Input: none. 

Output: returns the current state. 

Errors: None. Will always return a valid state. 



Description: request to the Server State Manager to change server 
state. This routine implements the guts of the state machine. 



ProcessRequest 



Omnishift Technologies, Inc. 



11 



Company Confidential 



eStream Server Component Framework Low Level Design 



1 . Get the current state, and transition request 

2. Index into the FSM table and continue to transition from state to 
state until the transition request is satisfied. 

3. Each state transition calls the specialized transition routines for 
each component. 

4. Call to SetState to complete each state transition. 

5. In the case of an error the state machine will process a 
RAISE_ERROR request which will call the specialized Han- 
dleError and transition to the ERROR state. 

Input: server transition request. Refer to table of valid requests de- 
fined above. 

Output: current state after the request has been completed. 
Errors: _____ 



Server Component Framework: 



class ServerComponent: ServerStateMgr{ // abstract base class 
private: 

Errorlnfo* Error; // maintains error if error was detected 
ServerConfig* Gonfig; // holds common configuration 
Connection* Listener; // messaging utility 

public: 

virtual int StartServer(void); // may be specialized by a server component 

virtual int StopServer(void); // may be specialized 

virtual int StartProcessing(void) = 0; // must be specialized 

virtual int StopProcessing(void) = 0; // must be specialized 

virtual int UpdateConfig(void) = 0; // must be specialized 

virtual int HandleError(void) = 0; // must be specialized 

void Run(Request); 
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StartServer 



Description: Called by the Server State Manager when a server compo- 
nent is to be started. The StartServer routine is provided as part of the 
SeverComponent class. It performs the following: 
1 . Send request to System Monitor to request an update of common con- 
figuration information. 

Apply the configuration information to the server component. 
Construct a listener connection object and start the message service. 
Return success or failure. 
Note: 

■ This routine must return immediately to the main thread. Otherwise 
the Server State Manager will be blocked. 

■ Successful return from the StartServer routine will put the server 
into the IDLE state. 

Input: None. 

Output: Value of 0 if successful else error condition 



Errors: May return negative error condition 



StopServer 



Description: Called by the Server State Manager. "~~ 

1 . Perform any necessary cleanup. 

2. Send last acknowledgment confirming shutdown to requester 

3. Shut down the messaging system and the listener. 

4. exit process 

Note: The monitor will update the database and perform logging. 
Input: None. 

Output: V alue of 0 if successful else error condition 

Errors: May return negative error value 
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StartProcessiog 



Description: Called by the Server State Manager. This routine must 
be defined by each specialized server component. This routine is 
used to provide all functionality unique to different types of servers. 
1 . Spawn a primary processing thread (also known as the boss 
thread). 

a. Read server specific configurations unique to this type of 
server component from the System Monitor 

b. Spawn worker threads. Depending on the server type this 
routine does the heavy lifting to either process access to- 
kens and renewals in the case of SLiM server, or process 
file requests for application servers, or manage and moni- 
tor the server components in the case of the System Moni- 
tor. 

Note: 

» This routine must return immediately so that the Server State 
Manager can continue to operate in the main thread. 

■ This routine may make use of the Server Configuration Manager 
for obtaining specialized configuration information 

Input: None 

Output: Value of 0 if successful else error condition. 
Errors: TBD 



Description: Called by the Server State Manager. This routine must 

be defined by the specialized server component type. 

1 . Reverse all actions performed by the StartProcessing routine. All 

worker threads should be joined or pooled in waiting state. 
Successful return from this routine will put the server component into 
the IDLE state. 
Input: None. 

Output: Value of 0 if successful else error condition. 
Errors: TBD 




UpdateConfig 



Description: Called by the Server State Manager. This routine must 
be defined by the specific server component type. The purpose of this 
routine is apply dynamic configurations or update specialized configu- 
rations that are unique to this server component. 
<may require adding a new state to separate dynamic and static con- 
figurations> 

output: v diuc Oi v xi aaccec>6iui else error coiium>jii. 
Errors: TBD 



HandleError 



Description: Component defined error handling routine to handle errors 
such as timeouts, etc. 

This routine will need to handle a number of error cases as are possible 
by the specialized component. The error information is maintained with 
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the ServerComponent class. 
Input: None. 

Output: Integer value designating a handled error or failure. If the error 
cannot be handled then it is fatal. 
Errors: TBD 



Run 



Description: This routine implements the main processing loop for the server 
component and runs in the main thread. This routine drives the server component 
by initiating state requests from the System Monitor. 
Note: The Server State Manager always runs in the main thread. 

1 . Call ProcessRequest to transition the server component into the initially re- 
quested state. 

2. Enter main processing loop 

a. Check for requests from the message service. 

b. Call ProcessRequest to service the request. 

c. Send acknowledgement for the request to the message service. Ac- 
knowledgement includes new state, load info, and status. 

Input: Initial Transition Request 

Output: None. This routine should never return 

Errors: None. 



Server Component Main Loop: 

The following main loop is common to all server components: 
void ServerComponent: :Run(ServerRequest Request) 

{ 

ProcessRequest(Request); 

while (1) 

{ 

Request = Listener->GetRequest(); 
ProcessRequest(Request); 

Listeners AckRequest(Request, GetState, GetLoad, GetStatus); 

} 



#inc)ude "Server.h" 

int rnain(int argc, char* argv[]) { 

Args = new ArgList(); 

Args->ProcessArgList(argv, argc); 
Server == new ServerComponent(GetValue(SID), 
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GetValue(DNS), 
GetValue(DBUSER), 
GetValue(PASSWD)); 
Server->Run(START_PROCESSING); 



Command Line Utilities: 



class NameValuePair 
{ 

private: 

char* Name; 
char* Value; 

public: 

NameValuePairO; 
-NameValuePairO; 
char* GetValue(void); 
char* GetName(void); 
char* SetName(char*); 
char*SetValue(char*); 

ii 



typedef int (*pFunc)(NameValuePair*); 

struct ArgTblEntry 
{ 

char* Name; 
bool Required; 
char* DefaultValue; 
pFunc ProcessFunction; 

ii : : 



ArgTblEntry const ServerArgsTblf] = { 



{"sid", 
{"dsn", 
{"dbuser", 
{"dbpasswd", 



true, 
true, 
true, 
true, 



0, 

0, 

0, 

0, 
n 



&ProcessSId>, 
&ProcessDsn}, 
&ProcessDbUser} , 
&ProcessDbPasswd } , 



typedef vector<NameValuePair*> ArgVector; 
class ArgList 

{ 

private: 
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/vrg v eciui 


Argvec, 


CUIJM /\Jgl UHMlliy 


/vrgiDi, 


pnvdie. 




NameValuePair* 


ParseArg(char* Arg); 


char* 


ParseName(char* Arg); 


char* 


x dibc v diucyynai txiQjy 


ini 


r rOCeSS/^Jg^lNdJilC V dlUCJTall )y 


int 


FinalizeArgs(void); 


public: 






ArgList(const ArgTblEntry*); 


int 

); 


ProcessArgList(char* argv[], int argc); 
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ProcessArgList 


Description: Process the entire argument list. 
In a loop for each argument argv[] ... 

1 . Call ParseArg passing in argv[]. 

2. ParseArg passes the result to ProcessArg 

3. After processing the entire argument list and exiting the loop call 
FinalizeArgs 

Input: argv and argc as passed into main() entry point 
Output: integer value designating success or failure 
Error: 


ParseArg 


Description: Takes a char* argument and verifies that it follows that 
name/value syntax defined as <name>=<value> 
Input: Next char* argument on the list 

Output: NameValuePair. NULL will be returned in the event of a syntax 

error 

Error: 


ProcessArg 


Description: This routine performs the semantic analysis of an argument. 

1 . Look up name in the ArgTbl 

2. Verify that the value is valid 

3. Add the name value pair to a list of processed arguments called ArgVec 
list. 

4. If this name value pair already exists in the list then issue a diagnostic. 

5. Call the supplied processing function for this argument as specified in the 
ArgTbl 

Input: NameValuePair 

Output: Interger value designating success or failure (0 for success, positive 

integer for other errors) 

Error: 


ParseName 


Description: Verify that the Name part of the argument conforms to being 
alpha-numeric 

Input: char* Name part of argument 
Output: char* Name else NULL 
Error: None 


ParseValue 


Description: Verify that the Value part of the argument conforms to being 
alpha-numeric and/or punctuation characters 
Input: char* Value part of argument 
Output: char* Value else NULL 
Error: None 


FinalizeArgs 


Description: Post process the argument list. The purpose of this routine is to 

r\isv pi OCc^oCo aiiU auUS uclatiifc cllgUiUCiiiS iU UiC r*lg V eC. 

Input: None 

Output: Success or Failure 
Error: 



Configuration Manager: 
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class Tuple { 

string name; 
Value value; 

}; 

class Value { 

int type; 

}; 

class StringValue: public Value{ 
string value; 

}; 

class TupleValue: public Value { 
vector <tuple> tupleArray; 

}; 

typedef vector < tuple > ConfigArray; 

class ServerConfig { 
private: 

ConfigArray Array; 

public: 

ServerConfig(serverId, dns, dbuser, dbpasswd); // Initialize from db 
ServerConfig(serverId, string filename); // To initialize from a file. 

ConfigArray* GetConfigArray(int serverld); 
Tuple* FindConfig(string Name); 
int Reload(void); 

Tuple* GetConfig(int serverld ,string Name); 

jj : 



ServerConfig 


Description: Constructor for Configuration Manager. 




1 . Initializes configuration manager. 




2. Opens the database and gets configuration array 




Input: Server Id, Data Source Name, Database User name, and database 




users password. 




Output: None 




Errors: 








1 . Initializes Configuration Manager. 




2. Opens configuration file and reads configuration array. 




Input: filename of flat-file configuration. 




Output: None 




Errors: 
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GetConfigArray 



Description: Returns the entire configuration for a given server id. 
This routine always retrieves its information either from the flat file 
or the database. 

Input: Serverld specifying which server to retrieve configuration for 
Output: Returns a vector holding the configuration or NULL 
Errors: 



GetConfig 



Description: Returns the configuration for the specified name. This rou- 
tine always retrieves its information either from the flat file or the database. 
Input: Serverld specifying the server to retrieve configuration for and 
Name of configuration item. 

Output: Configuration Tuple. A Tuple may be a nested Tuple. NULL if 

an error is encountered. 

Errors: 



FindConfig 



Description: Returns the Tuple specified by the name. This routine does 
not go to the database or flat-file to get its value. Rather it finds the value 
in the ConfigArray maintained by the Configuration Manager. 
Input: Name of the configuration item. 

Output: Configuration Tuple. NULL if an error is encountered or the Tu- 
ple does not exist in the current configuration. 
Errors: 



Reload 



Description: Reloads the entire configuration from the database or flat- 
file. This routine may reload its configuration indirectly through the use of 
the System Monitor. In this case it will make a message request to the 
monitor and listen for the configuration results. 
Input: None 

Output: integer specifying success or failure. Zero will be returned in the 

case of Success. A negative value in case of error. 

Errors: 



Logging Utilities: 



class LogManager 

{ 

private: 

char* FileName; 

cnar * Kesourceriie; // message catalog tile 
char* GetMessage(MsgNum, MsgStr) 



public: 



LogManager(ServerId,Size=10); 
LogMessage(MsgStr); 

LogMessage(ThreadId, MsgNurn, MsgStr, ...); 
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}; 



LogMessage 



Description: Write message out to log file. There are two forms of 
LogMessage. The first will write out a message buffer as is (unformatted) 
bypassing the resource file. 

The second form will format the message. Both forms of LogMessage 
always pre-append a time stamp. 

1 . Lookup message number in the resource file and get message string 

2. format the log message using time stamp, thread id, etc. 

3. write out message into the log file. 

Input: Thread Id, Message Number, Message String, and variable num- 
ber of arguments. 
Output: None. 
Error: 



GetMessage 



Description: Routine returns a message string from the resource file for 

the message number specified. 

Input: Message number, C Locale text string. 

Output: Message string. Either way, Get Message will always pass a • 
return a valid message string by either returning the string from the re- 
source file or by passing back the MsgStr passed in. 
Error: If an error occurs trying to get a message from the resource file, a 
message will be logged to the error log. 



class ErrorLog: protected LogManager 

{ 

private: 

LogLevel ErrorLogLevel; 

public: 

ErrorLog(ServerId, LogLevel=0, Size=10); 
LogError(ThreadId, ErrorNum, ErrorMsgStr, . . 

jj 
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LogError 


Description: Writes output to error log file. 




1 . Check that the message level against the current ErrorLogLevel. 




2. Format the message and call the lone form of LoeMessape to write the 




buffer out to the file 




Input: 




1 . Threadld: Thread identifier to helo with the debueeinp nroce^ 




z.. i^i i uj i>» uin. i^nuj uuiiiuci uocu iu uiiii|ucijr juciiiiiy aii error message in 




the resource file 

Vll V 1 vuvUl vv ill v » 




3. ErrorMsgStr: Message string which includes stdio like string formatting. 




4. . . . : variable list of arguments to be inserted into the message string per 




the format. 




Output: None. 




Error: 



Testing design 



Each of the components that make up the Server Component Framework will be able to 
be tested independently of the other components. Each component will have a main en- 
try point defined within a testing .exe to accomplish the Unit testing phase. 



Testing of the component framework will be done in phases. Each of the phases is de- 
scribed below along with its dependencies. 



Phase l:Unit testing 

Test basic components that make up the frame- 
work. Each components functionality, restric- 
tions, and boundary conditions will be tested. 

Will allow testing common configurations for a 
single server component. This round of unit test- 
ing will test the integrated component utilities 
and framework. 


Dependencies: 

1 . ServerComponent class 

2. ServerStateMgr class 

3. ArgList class 

4. Logging Utilities 

5. Configuration Manager (flat-file) 


Phase 2: Unit testing (full functionality) 
Test full functionality including messaging inter- 
faces and database connectivity. 


Dependencies: 

1. Phase 1 

2. Database connectivity 

3. Messaging Service 


Phase 3: Integration Testing 


Dependencies: 

1. Phase 2 

backup) 

3. SLiM Server, App Server, Web- 
Server 


Phase 4: Stress Testing 

See section on stress testing for details 


Dependencies: 

1. Phase 3 
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Unit testing plans 

Command Line Utilities 

The Command line utilities will be tested in a stand-alone module called cmdline.exe. It 
will support the command line arguments defined in this document. 

Configuration Manager 



The configuration module is a stand-alone module which will be tested using a config- 
test.exe executable. The executable will exercise all of the interfaces described above. 
The configtest.exe executable should be testable in the DB and the non-DB mode. 



Logging Utilities 



The logging utility will be built as a DLL (otlog.dll). We will provide a binary otlog- 
test.exe which will exercise each of the interfaces mentioned above. 



Server State Manager 

The Server State Manager and the Server Component Framework will be tested inde- 
pendently of specialized components. The routines that require specialization (Start- 
Processing, StopProcessing, HandleError and UpdateConfig) will be provided to sim- 
ply return successfully. 



Stress testing plans 



Stress testing will require having at least the System Monitor functionality implemented 
since it is used to drive the server components. 



1 . Test to repeatedly start, stop, reconfigure the server component. 

2. Test to crash machines with server components to validate: 

a. data persistence. 

b. detection capabilities and response. 

c. auto restart. 

3. Test to kill individual server component processes. 

a. data persistence. 

b. detection capabilities and response. 



4. Test lost database connectivity 

5. Test lost of messaging capabilities 

a. repeatedly losing and re-establishing messaging connectivity 

6. Test error recovery under adverse conditions. 

7. Test recovery from running out of memory, thread resources. 

8. Test recovery from threads dying. 

9. etc. 
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Coverage testing plans 

1 . Goal: 100% path flow coverage. Only exceptions for known error conditions that 
cannot be practically reached (e.g. thread synchronization, etc.) 

Cross-component testing plans 

The following pair-wise testing will be performed: 

1 . framework/database (phase 2) 

2. framework/messaging (phase 2) 

3. framework (System Monitor) /framework (backup Monitor) (phase 3) 

4. framework/Web Server (phase 3) 

5. framework (System Monitor) /framework (Other Servers) (phase 3) 

Upgrading/Supportability/Deployment design 

1 . Each error condition will be documented with explanations and practical work- 
arounds 

2. Component framework will support enhanced debug option to dump additional de- 
bugging information to special log files. 

Open Issues 
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Functionality 

The role of the System Monitor is to monitor the state of the Application Servers and 
SLiM servers within an eStream deployment. In addition, it also manages a back-up 
System Monitor. 

□ The System Monitor provides the following key services: 

a. Monitors and reports server load across machines 

b. Monitors server state across machines. 

c. Acts as a communication conduit to the database for configuration 
information needed by the server components. 

d. Initiates state changes within the server. 

i. Start/Stop servers 

ii. Sends requests for servers to update their configurations 

□ The system monitor runs as it's own process. Within a multi-system deployment, 
there will be at least two monitoring processes, each on a different machine: 

a. One monitoring process will act as a primary and the others as backups. 

b. In the event that the primary monitor goes down, one of the backups will take 
over the primary monitoring responsibilities. 

□ The monitoring process can re-launch a server side process if a process terminates 
unexpectedly (this is a configurable option). 

□ The system monitor manages server state by maintaining regular communication via 
a heart beat protocol between itself, the backup monitors, and every logical server. 

□ The monitor will raise an alarm if it does not receive a heart beat response from the 
servers within a specified period of time. 

□ The system monitor's heart beat request rate is a dynamically configurable parameter 
maintained within the database. 

a. The rate can be changed through the administrative interface. 

□ The system monitor's heart beat supports a light-weight messaging protocol between 
components to initiate state changes. 

a. Simple request pulse. 

b. Stop request pulse 

c. Configuration request pulse 



The System Monitor process is extended from the Server Component Framework by 
implementing the plug-able interfaces which includes: 



StartProcessing 



StopProcessing 

UpdateConfig 

HandleError 



System Monitor States: 



When the System Monitor transitions from the STOPPED state to the IDLE state it is 
maintaining only its main thread which it inherits from the Server Component 
Framework and runs the Server State Manager. The messaging Service may or may not 
maintain its own threads. For the purpose of this design we will assume that it is 
transparent to the System Monitor 




When the System Monitor transitions into the PROCESSING state it employs a "Boss- 
Worker" parallel programming model: 

1 . The System Monitor creates a processing thread that acts as the boss thread. 

2. The boss thread reads the configuration for all server components and spawns worker 
threads for each logical server component. 

3. Worker threads are allocated, in turn, by the boss to start, monitor, and manage each 
server process running within a deployment. 

4. The boss thread spawns a special thread to handle configuration requests from all the 
server components. 

5. The Load Monitor runs as a service within the boss thread. 



Synchronization between worker threads and the boss thread is provided through request 
queues. 

Load Monitor: 

The Load Monitor's role is to aggregate the load information for each server component 
within the deployment and update aggregated information to the database. 

The Load Monitor runs in the "boss" processing thread of the System Monitor. 

For each application the Load Monitor maintains a list of servers and their response time. 
It periodically updates the database with the order lists which are consumed by the SLiM 
servers. The frequency at which it updates the database is configurable. 

The Load Monitor maintains three interfaces: 



UpdateLoad 


Update in memory server sets with the current load 
information passed in as an argument. This routine is called 

f kOCESSunG state. | 


DbGetServerSets 


Retrieve current server sets in the event that System Monitor 
crashes and/or starting System Monitor. 


DbSetServerSets 


Update database with lists of servers for each app 



Data type definitions 



The System Monitor maintains a few key data structures beyond what is provided by the 

Server Component Framework: 

Request Queue: 

Each worker thread will maintain its own mutex protected input queue. The input queues 
are used primarily to communicate requests between threads. A queue entry models, 
fairly closely, the data contained within the messaging protocol between servers as 
defined in the Server Component Framework. This data can be used for the input queues 
for each worker thread managing individual work components. 

The details of the input queues themselves is defined in the common thread API. 



struct QueueEntry { 
int Request 
ServerlD SenderlD 
ServerlD Recieverff) 
ServerlD TargetID 
Data 

j : 

Managing Worker Threads: 

The System Monitor's boss thread maintains an in memory list of all known servers 
within a deployment, and their associated worker thread ID. In addition, it maintains 
information about request queues for each worker thread. This list is maintained as the 
SystemMonitorList. The list is made up of entries which maintains all the necessary 
information to start/stop/manage/communicate/etc. with a server component. 



SystemMonitorEntry = struct { 

ServerConfig* Serverlnfo; 
Thread ThreadID; 
ThreadQueue* InputQueue; 

> 

typedef vector <SystemMonitorEntry> SystemMonitorList; 



iiiittiicii;*? ueiiiiiClOliS 

The implementation of the System Monitor is derived from the generic ServerComponent 
class. Refer to the Server Component Framework Low Level Design. 

Implementing a SeverComponent requires the definition of the following methods: 



StartServer 


Description: This routine is called to start a server process and 
bring it to the IDLE state. It assumes that the server is in the 
STOPPED state. This routine performs all the necessary 
initialization and configuration common to all server components. 

This function is predefined in the ServerComponent base class. 
The System Monitor has no need to override it. 

Input: None. 

Output: Integer value designating return status. Success = 0. 
Errors: 




StopServer 


Description: The routine StopServer is called to cleanup and 
terminate a server process. It assumes that the server component is 
in its IDLE state and is therefore not processing any requests. 

This function is predefined in the ServerComponent base class. 
The System Monitor has no need to override it. 

Input: None. 

Output: Integer value designating return status. Success = 0. 
Errors: 



StartProcessing 


Description: This routine initiates all the activities unique to the 




System Monitor: 




1 . Launches a boss thread. Call MonitorServers as the boss 




thread entry point. 




2. Return immediately 




Note: The StartProcessing routine must return immediately else 




the Server State Manager will be blocked. 




Input: None. 




Output: Integer value designating return status. 




Errors: 



StopProcessing 


Description: This routine will perform the following activities: 

1 . Terminates each server component's monitoring thread. This 
effectively disables the monitor from managing any executing 
server components including stopping existing components or 
starting new ones or servicing configuration requests. 

2. Disables the load balance manager. 

Note: The primary system monitor will only execute this interface 
if a system administrator explicitly changes the monitors state or an 
error has occurred within the monitor. 
Input: None. 

Output: Integer value designating return status. 
Errors: 




UpdateConfig 


Description: This routine will perform configuration changes 
specific to system monitor functionality. 

Input: None. 

Output: 

Errors: 



MonitorServers 



Description: This routine is the thread entry point for the boss thread. 
It is the primary thread for managing and monitoring all the server 
components. The Load Manager runs in the boss thread. 

1 . Launch the configuration management within its own thread. 

2 . Construct/ini ti alize Load Manager 

3. Go to the database and get a list of all the server components. 

4. For each server component retrieve its common configurations 
and create and entry into the SystemMonitorList 

5. Spawn a worker thread to manage/monitor the backup System 
Monitor. Call MonitorServer as the thread entry point passing in 
the SystemMonitorEntry After the backup monitor has been 
launched, repeat step 4 for each server component. 

6. Loop back and redo steps 2-5 looking for new servers to manage 
Input: 

Output: 
Errors: 



MonitorServer 


Description: Launches and monitors the specified server component 




process. It can launch server process' locally and on remote 




machines. It is expected to be the start routine for a new thread. 




Performs the following steps: 




1. 


Call StartServerProcess to launch the server component per the 






SystemMonitorEntry. 




2. 


Open a point-to-point messaging connection to the newly 






launched server component. 




3. 


Initiate a regular heart-beat request. 




4. 


Enter monitoring loop 






a. Check for requests from the worker thread input queue 






b. Process requests by sending request to the server currently 






being monitored by this thread. 






c. Listen and wait for response; timeout if wait is too long. If 






there is a timeout exit thread with an error and let the error 






handler deal with it. 






d. If a response is received update state information in 






SystemMonitorEntry. 






e. Enqueue load information onto Load Balance managers 






input queue. 






f. repeat a-f 




Input: SystemMonitorEntry. 




Output: Status of the server component on exit- 




Errors: 




1. 


launch error 




2. 


messaging error 




3. 


unexpected server component termination 



Primary and Backup System Monitor: 

The backup System Monitor needs to be able to take over the primary system monitor 
responsibilities in the event that the backup loses communication with the primary. The 
following interface will cause the backup monitor to take over as primary. 



SwitchPrimarylVlonitor 



Description: Take over primary monitor responsibilities. 

1 . Update database validating monitor switch. 

2. Shut down the old primary just in case it is still running 
using by opening a connection to the old primary and 
sending it a STOP_SERVER request. 

3. Go through the same steps as the primary monitor would 
do when its StartProcessing routine is called including 
starting up a backup system monitor 

Note: It may be more appropriate to define an error handler 
that is called when the heart beat is lost. The error handler 
would go through steps 1 and 2 and then request a state 
change which would result in the monitor coming on line as a 
primary monitor. This requires more thought but would be a 
more elegant solution. 
Input: None 

Output: integer value designating success or failure 
Error: 



StartServerProcess 


Description: This routine spawns the requested process either on 




the local machine or on a remote machine. 




1 . Verifies that the target machine is up and running. 




2. Launch process specified according to attributes 




Input: Process name, target machine, attributes 




Output: integer return where 0 designates success else error code 




is returned 




Errors: 




1 . machine not responding 




2. executable not found 




3. failure on launch 



Component design 
Testing design 

The System Monitor will provide a command line option to facilitate testing the 
component in its various testing phases. The option will allow the system to manage its 



Unit testing plans 

The System Monitor can be unit tested in the following phases: 



Phase 1: 

1 . Test state management and messaging 
capabilities of all server components 

2. Test starting, stopping, updating configs 


Dependencies: 

1 . Common Server Framework 

2. Messaging 


Phase 2: 

1 . Test data persistence and error recovery 

2. Test config updates from database 

3. Test back-up monitor and switch-over 


Dependencies: 

1. Phase 1 

2. Database Connectivity 


Phase 3: 

1 . Test load balance manager and components 
sending load information 


Dependencies: 

1. Phase 1 &2 

2. Load Balance Manager 


Phase 4: 

1 . Full functionality testing 

2. Stress testing 

3 . Full error recovery testing 


Dependencies: 

1. Phase 1, 2, & 3 

2. Webserver 

3. App Server/Slim Server 



Stress testing plans 

Stress testing will be performed at Phase 4 testing. Tests should include: 

1 . Max # of generic server components on a single machine. 

2. Max # of generic components across multiple machines. 

3. Max # of generic servers changing state at least once per second 

4. Test to repeatedly start, stop, reconfigure each server component. 

5. Test to crash machines with server components to validate: 

a. data persistence. 

b. detection capabilities and response. 

c. Auto Restart. 

6. Test to kill individual server component processes. 

a. data persistence. 

b. detection capabilities and response. 

c. Auto Restart. 

7. Test lost database connectivity 

8. Test lost of messaging capabilities 

a. repeatedly losing and re-establishing messaging connectivity 

9. Test error recovery under adverse conditions. 

10. Test recovery from running out of memory, thread resources. 



Coverage testing plans 



The System Monitor will achieve 100% code coverage with the exception of error 
conditions which are possible however difficult to reach in practice. 



Cross-component testing plans 



The System Monitor interacts with the following components: 

1 . System Monitor/Database 

2. System MonitorAVebServer 

3. System Monitor/Server Component Framework (other servers) 

Upgrading/Supportability/Deployment design 

1 . All diagnostics will be documented as to their root cause and workarounds/actions to 
be taken. 

2. The system monitor will support an enhanced debug support which dumps additional 
information to special debug logs. 

Open Issues 

1 . Need to figure out how to launch a process on a remote machine. May need to use 
the slave monitor to actually launch the process. But then the question arises . . . how 
does one launch the backup monitor? 

2. Need to identify a mechanism to ensure that we do not have more than one primary 
monitor running at any given time. 

3. Need to strike a balance on how many threads to spawn. All the server components 
may be able to be monitored and managed out of a single primary thread which is 
referred to as the boss thread. 
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eStream Web Server/Database Low Level Design 



Bhaven Avalani 




Functionality 

The eStream solution provides a set of account, user, and subscription manage- 
ment utilities. These utilities are provided as extensions to the ASP's(Application 
Service Provider) web server. 

There are three categories of users for these utilities: End User, Group Adminis- 
trator and ASP Administrator. The roles and the capabilities of each of these us- 
ers are detailed below. 

End user for a system is the user who will actually access eStream application us- 
ing the eStream clients. An end user should be able to: 

• Create Account and User attributes. (Username, Password, etc.) 

• Change Account and User attributes. 

• View all available applications in the eStream system. 

• Subscribe/Manage eStream applications. 

• View Account Status. 

1. List of applications subscribed. 

2. Status of current subscription. 

3. View/Change Billing information. 

4. View/Change Account information. 

A Group Administrator is an administrator for a group of users. An individual 
user is by definition a group administrator for a single user group. Capabilities of 
a group administrator are: 

• (All of single user capabilities). 

• Add delete users from a group. 

• Manage the active sessions for a group. A group manager should be able 

• View the billing information. This will probably need hooks to an external 
billing system. 

An ASP administrator manages the overall application system. Capabilities of an 
ASP administrator are: 

• Manage accounts/users/subscription for all users/groups in the system. 

• Manage the application data for a subscription system. 
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1 . Add new applications to the system. 

2. Modify application information for the system. 

3. Provide the pricing mechanism for the applications^). 
• Manage the servers in the system. 

1 . Configure a server. 

2. Stop/Start a server. This is accomplished by a message to the 
Monitor server. 

3. Get load information for a server. 

4. Get logging information for a server. 



There are essentially two different types of accounts, which the system will support: Sin- 
gle user account and corporate accounts. 

The following licensing mechanisms will be supported by the system. 

• Fixed Duration License. (Typically monthly license). 

• Fixed Duration Floating License. An example of this is n licenses for k users for a 
fixed duration. 

• Indefinite License. 



Description 

There are several key issues that need to be determined for the Web Server architecture. 
The options available in the market to implement these technologies are listed below. 

Web Server: 



• Apache 

• Netscape Server 

• Microsoft Internet Information Server 
CGI Technology 

• Servlet/JSP 

o Tomcat ( from Apache group) 
o JRun (from Allaire) 

• iNSAH ( C level API available for Netscape and Apache). 

• ISAPI ( C level API available for IIS and Apache) 

• CGI (Perl/C etc.). 



Database Connectivity 
• JDBC. 
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• ODBC 

• Native. 

Database 

• SQLServer 

• Oracle 

• Sybase 

• Informix 

• LDAP(??) 

The overall proposed solution for eStream 1.0 Webserver release is: 

Apache + Tomcat(for JSP/Servlet) + JDBC + SQLServer. 

The reasons for choosing this combination for the servers are as follows: 

1 . JSP/Servlet is the only technology which is available for cross-platform and cross 
Webserver support. 

2. We need to decide on a single web server to develop and test against for release 

1 .0. Apache is chosen to be the one as it is popular on Unix and NT platforms and 
it is freely available. 

3. Tomcat(Apache group's reference implementation for JSP/Servlet specs) is the 
preferred CGI technology as it works well with Apache and all other web servers. 

4. JDBC is preferred for database connectivity as its database neutral and works well 
with Java environment of Servlets. 

5. SQLServer is the preferred database for release 1 .0. This contains the scope for 
testing and deployment for eStream 1.0. 

Since all other servers(App Server, SLM Server and Monitor) are C++ components, the 
following technology combination will be available for Database Access. 

ODBC + SQLServer. 



The data model for the eStream 1.0 database essentially consists of two high level com- 
ponents. The database deployment architecture is shown below: 
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ASP ADMIN 




Server Management Component: This component's primary responsibility is to man- 
age the configuration, load and log information for a logical server in the system. The 
clients to this component are all the servers and administration manager. A detailed list of 
interfaces for this component is described in the interfaces section. 

User/Account/Subscription Management: This component is responsible for maintain- 
ing the user account and subscription information for the system. The end user using the 
end user interface performs the updates to this component. Slim Server will access this 
component to validate subscriptions. A detailed list of interfaces for this component is 
described in the interfaces section. 

Group Management: This component is useful for managing groups of users. The group 
administrator can only perform updates to this component. A detailed list of interfaces for 
this component is described in the interfaces section. 

Billing Management: This component's responsibility is to provide interfaces to an ex- 
ternal billing system. A detailed list of interfaces for this component is described in the 
interfaces section. 
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Application Management: This component's responsibility is to provide application 
management interface. This component is accessible for updates only by the ASP admin- 
istrator. A detailed list of interfaces for this component is described in the interfaces sec- 
tion. 

License Manager: This component's responsibility is to manage the licenses. SLiM 
server will check out licenses from the license manager. A detailed list of interfaces for 
this component is described in the interfaces section. 



The architecture for the Web Server extensions implementation is shown below: 




Worker Bean 



Worker Bean 



Data Access 
Bean 



The basic elements of this architecture are as follows: 



1. i^very request into the system goes tarougn a dispatcher servlet. This servlet will 
perform initialization, initial validation of the request and miscellaneous checks 
before dispatching the request to a JSP page. A worker bean will responsible for 
performing the initialization. The processing of the incoming request is performed 
at this stage. The request is then dispatched to an appropriate JSP page. 

2. The JSP page will invoke worker beans to access the dynamic data from the data- 
base via the Data Access Bean and the resultant page is sent back to the user. 
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This architecture is illustrated with the following example. 

1 . User sends in a request to update the usemame and password information in the 
database. Inputs are username, old password, new password. 

2. The dispatch bean will call the user(worker) bean to: 

a. Validate the user's old password. 

i. The user worker bean will make a request to the data access bean 

to access the password for the user, 
ii. The two passwords are compared and the result is returned. 

b. If the password was valid then, update the new password. 

i. Call the data access bean to update the password in the database. 

c. Else return failure. 

3. Based on the success or failure the dispatcher will dispatch the page request to the 
appropriate JSP page. (eg. error.jsp on failure and user.jsp on success). 

4. The page will invoke the appropriate the worker bean (error bean or user bean) to 
obtain the dynamic data and send the response back to the user. 

The salient features of this architecture are: 

1 . Presentation and processing logic is separate. Thus, the customer(ASP) can cus- 
tomize the look and the feel of the pages without impacting the processing logic 
as it is segregated. 

2. The data access bean is separated from the worker beans, which are primarily re- 
sponsible for the business logic. This allows us to change the data access layer (eg 
enabling LDAP access) in the future without impacting the system drastically. 



Data type definitions 



The central data structure for Web Server is the database model. The overall database 
model for user and subscription management is shown below. 
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PK 


Usaqeitf 


FK1 


starttime 
duration 
subscriptionld 




AppName 
Version 

Description 







PK 


Accpyntltf 




AccountName 




Address 1 




Address2 




City | 




State 




Zipcode 




Phone 1 




Ext1 ! 




Phone2 




Ext2 




Fax 




cardType 




cardNumber 




cardExpiry 




cardFirstName 





PK 


SMtecnptfonltf 




FK4 


Group Id 


FK3 


Userld 




createdate 




modifydate 


FK1 


Applicationtd 


FK2 


Licenseld 








PK 


Mcensetf 




createdate 

modifydate 

expirydate 

maxusers 

available 







PK 


Userld 


FK1 


Groupld 


11 


username 




password 




usertype 




notes 


FK2 


Accountld 



1 





cardMiddlelnitial 




cardBillingAddrl 




cardBillingAddr2 




cardCity 




cardState 




cardZipcode 







FK1 


Userld 
groupname 

notes 
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The important features of this data model are: 

Account: Table holding all the billing and contact information for a user or a group. 

User: An end user in the system. A user can optionally belong to a group. 

Group: A group of users. One of the users in the group is designated as the group admin- 
istrator. Each group has a unique account associated with it. 

Application: This table contains the data about various applications in the supported by 
the ASP. 

License: Each row in this table corresponds to the licensing term for a given subscription. 
This table also maintains the active count of the licenses checked out. 

Subscription: This table contains entries for subscription items. A subscription item con- 
sists of user/group, application and license. 

Usage: This table contains the runtime information for a system. SLiM server updates 
this table with access token usage data. A billing system may interface with this table to 
generate billing data. A reporting system may interface with this table to report on usage 
patterns. 

LicenseUsage: This table is responsible for recording checked out licenses in the system. 

The data model for storing the server related information is shown below: 

PK: Primary Key for the table. 

FK: Foreign Key. Used for relations between tables. 

11,12.. : Index Columns. 
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The tables in this model are: 

Server: This table contains entries for each logical server in the system. 

Machine: This table contains entries for each physical server in the system 

Configuration: This table contains configuration entries for a given server. The configu- 
ration entries can be hierarchical in nature. Each configuration has the following format: 

Name Valuel [Value2] [Value3] [ParentConfigld] 

Load: This table maintains the historical and real-time load information for a given logi- 
cal server in the system. 

Log: This table maintains the logs for a logical server in the system. The log messages 
saved here are "major" events in the logical server system. A detailed logs stored in a flat 
file on the physical machine containing the logical servers. 

Global Data Structures: 

struct oerveri upie 
{ 

int serverld, 
int type, 

String serverName 

}; 
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struct Couple 

{ 

String name, 
String value 

}; 

For the Access Token and related data structures, please refer to the SLiM server Low 
Level Design Document. The interfaces below will discuss some of the API's based on 
the these data structures. 



Interface definitions 

The interfaces exposed by various sub-components are detailed below. 

Server Management Component: 

CreateServer 

int CreateServer (ServerConfig* config) 

Input : 

Server Configuration. 
Output: 

Unique ID for the server. 
Commen ts : 

Create a logical server with predefined con- 
figuration . 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

UpdateServerConfig 

Bool UpdateServerConfig(int serverld, String name, String value) 

Server Id 

Config name and value 
Output: 

Unique ID for the server. 
Comments : 

Create a logical server with predefined con- 
figuration. 
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Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

AddMachine 

Bool AddMachine(String name, String domain, String ip) 
Inpu t : 

Machine name domain and ip. 
Ou tpu t : 

Success/Fai lure 
Comments : 

Create a physical machine entry. 
Errors : 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

SetServerLog 

Bool SetServerLog(int serverld, LogTuple log) 

Inpu t : 

Server Id 

Log tuple (data structure in the Logging 
document . ) 
Ou tpu t : 

Success/Failure 
Comments: 

Add the log data for a server 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerLog 

LogTuple[] GetServerLog(int serverld, int maxrows = 25) 

Jnput: 

Server Id 

Maxrows: Maximum number of rows to be re- 
turned . 
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Output: 

Array of Log tuples (data structure in the 
Logging document . ) 
Comments : 

Get the log data for a server 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServers 

Serverf] GetServersO 
Input: 
Output: 

Array of Server tuples (data structure de- 
fined above) 
Comments: 

Get all the server information 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

SetServerState 

Bool SetServerState (int serverld, short state) 
Jnpu t : 

Serverld: Unique id for a server 
State: State information for a server. 
Output: . 

Bool True/False for success/failure. 
Comments : 

Update the database with current state in- 

Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 
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GetServerState: Obtain the last known state for a specified server 
short GetServerState (int serverld) 
Input : 

Serverld: Unique id for a server 
Ou tpu t : 

State: State information for a server. 
Comments: 

Obtain the last known state for a specified 
server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerConfig: Obtain configuration information for a specified server 
ServerConfig* GetServerConfig (int serverld) 
Inpu t : 

Serverld: Unique id for a server 
Output: 

ServerConf ig* : State information for a 
server. (ServerConfig data structure is defined 
in the server configuration document) . 
Comments: 

Obtain the last known state for a specified 
server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



SetLoadData: 

void SetLoadData (int serverld, int Load) 



Inpu t : 

Serverld: Unique id for a server 
Load: Load for the server 
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Output: 
Comments: 

Monitor may call this interface to persis- 
tently store historical load data. It is still 
not clear if SLM and application servers will 
store this directly themselves. 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerLoad: 

void GetServerLoad (int serverld, , int maxrows = 25, int** Load) 
Jnput : 

Serverld: Unique id for a server 
maxrows: Maximum number of rows to be re- 
turned .Default is 25. 
Output: 

Load: Load for the server 
Comments: 

Obtain server component load information to 
manage load balance . 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



FIushLoadData: 

void FIushLoadData (<tuples> LoadData) 



LoadData tuples containing <server id, 
server load> values. 
Output : 

Comments: 

Used to flush aggregated load data to the 
databa 
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Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

User/Account/Subscription Management Component 

CreateUser. This API is used to create user record in the system. Arguments will be 
Usemame, Password. 

Bool CreateUser(String username, String password) 

ValidateUsen This API is used to validate user record in the system. Arguments will be 
Usemame, Password. 

Bool ValidateUser(String username, String password) 

CreateAccount This API is used to create account records in the system. Arguments 
will be billing address, credit card information etc. 

Bool CreateAccount(String usemame, <Account Information>couple[]) 
Input: 

Usemame associated with the account. 

An array of names and values for the account. 
AddSobscription. This API is used by the end users/group administrators to subscribe to 
applications. 

Bool AddSubscription(<Subscription Information>couple[]) 

Input: An array of names and values for the subscription. 



UpdatePassword. Used to change user information. Password, usemame etc. 

Bool UpdatePassword(String username, String old-password, String new- 
password); 

UpdateAccount Used to update the account information. Billing Address etc. 

Input: An array of name, value pairs for the fields to be updated. 
UpdateSubscription. Used to add additional time to a subscription. 
Bool UpdateAccount(Couple[]) 
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Input: An array of name, value pairs for the fields to be updated. 

GetUserRecord. Used to get current user configuration. 

Couple[] GetUserRecord (String username) 
GetAccountRecord. Used to get current account configuration for a user. 

Couple[] GetAccountRecord(String username) 

GetSubscriptionRecords. Used to get to subscription records in a database. End user 
may just want to verify what they are subscribed to. 

Couple[][] GetSubscriptionRecords(String username) 

Output: Ah array of array of couples containing the subscription informa- 
tion for a given user. 

DeleteUser. Used to delete users who are no longer valid in the system. Typically called 
by the ASP admin. 

Bool DeleteUser(String username) 
DeleteAccount. Used to delete un-used accounts. 

Bool DeleteAccount(int accountld) 
DeleteSubscription. Used by the ASP admin to remove subscriptions. 

Bool DeleteSubscription(int subscription^) 

Group Management Component 

Createljiroup. i Mis APJ is responsible ior creating group accounts in the database. 
Called by the group admin user. 

Bool CreateGroup(String groupName, String admin, String notes) 
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AddUserToGroup. Adds a user to a group, 

Bool AddUserToGroup(String groupName, String username) 
DeleteUserFromGroup. Removes a user from a group. 

Bool DeleteUserFromGroup(String groupName, String username) 



GetActiveSessions. Gets the active sessions for a group. 

Couple[][] GetActiveSessions(String groupName) 

Output : An array of array of couples containing the following information 
for each active session in the system: 

Username 

Licenseld 

StartTime 

EndTime 

Subscription 



Licensing Component 



CheckoutLicense: Checks out a license. 

int CheckOutLicense(int subscription^, long* pStartTime, long* pStopTime) 

Inputs: 

Subscriptionld: Subscription id of the user. 
Outputs: 

>0 for a successful license. The output is the license usage id. 
StartTime for the license. 
StopTime for the license. 

x we &>atoni aiiouiu validate cue avaiiaoiiuy oi me iicense. 

Errors: 

INVALID USER ID 
INVALID SUBSCRIPTION 
LICENSE NOT AVAILABLE 
NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 
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RefreshLicense: Refreshes a license. 

int RefreshLicense(int LicenseUsageld, long* pStartTime, long* pStopTime) 
Inputs: 

LicenseUsageld: License usage id. 
Outputs: 

>0 for a successful license. The output is the license usage id. 
StartTime for the license. 
StopTime for the license. 
Comments: 

The system should validate the availability of the license. 

Errors: 

INVALID USER ID 
INVALID SUBSCRIPTION 
LICENSE NOT AVAILABLE 
NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 
EVICTION 



CheckinLicense: Check in a license 

Bool CheckInLicense(String username, int subscriptionld, int licenseUsageld) 
Inputs: 

Username: user trying to check out the license 

Subscriptionld: Subscription id of the user. 

LicenseUsageld: Usage id for the checked out license. 
Outputs: 

Success/Failure 
Comments: 
Errors: 

INVALID SUBSCRIPTION 
NO DATABASE CONNECTION 
UNKNOWN SQL . ERROR 



Valid ateLicense: Validate that the user has a license checked out. 

Bool ValidateLicense(String username, int subscriptionld, int licen- 
seUsageld ) 

Inputs: 
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Username: user trying to check out the license 

Subscriptionld: Subscription id of the user. 

LicenseUsageld: Usage id for the checked out license. 
Outputs: 

Yes/No. 
Comments: 
Errors: 

INVALID USER 

INVALID SUBSCRIPTION 

INVALID LICENSE 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



DBAcquireAccessToken 

RPCReturnCodes DBAcquireAccessToken(long Subscriptionld, long* pAccessTokenld, 
string UserName, string Password, long* pStartTime, long* pStopTime, long* Applica- 



tionld) 






IN 


Subscriptionld 


Id of the subscription being used. 


IN/OUT 


pAccessTokenld 


-1 if this is a first time access. 


IN 


UserName 


Username string. 


IN 


PassWord 


Encrypted Password 


OUT 


pStartTime 


Start time for Access Token validity. 


OUT 


pStopTime 


Stop time for Access Token validity. 


IN/OUT 


Applicationld 


Id of the application. -1 Default. 


OUT 


RPCReturnCodes 


RPC Return codes. 


Processing: 







This is fairly complex function. The processing involved in this function call is: 

• If this is the first access (ie *pAccessTokenId = -1) then ValidateUser 

• If the Applicationld is -1 then GetAppId 

• If this is the first access (ie *pAccessTokenId == -1) then CheckoutLicense 

• If this is a renewal request: RefreshLicense 

• If there is a failure and it is due to eviction: GetEvictionReason 



#define RPCRUSERAUTHFAILED 

#define RPCR_ACCESS_TOKEN_INVALID 

#define RPCR_ACCESS_TOKEN_EXPIRED 

#defme RPCRLICENSENOTAVAILABLE 

#define RPCRLICENSEALREADYHELD 

#define RPCR_EVICTION_NOTICE 
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#define RPCR^EVICTION__MUST_^UPGRADE 
#define RPCREVICTIONENDMEMBERSHIP 
#define RPCRJBVICTIONJ^O_PAYMENT 



DBReleaseAccessToken 
DBReleaseAccessToken(long AccessTokenld) 

IN AccessTokenld 

Processing: 

• Update the Usage table with the appropriate information. 

• Delete the LicenseUsage record. 
Notes: 

• We need a mechanism to release un-released access tokens. The way to do this 
would be to run a stored procedure at demand and at a predefined intervals to do 
this cleaning up. 

EvictAccessToken 

DBReleaseAccessToken(long AccessTokenld) 
IN AccessTokenld 

Processing: 

• Evicts an access token. 



Billing Component 

AddUsageRecord. Called by the SLM server when it releases an access token. 

Bool AddUsageRecord(String username, int subscription^, date starttime, long 
duration). 

GetUsageRecordsForUser. Used by external billing system. 

GetUsageRecordsForGroup Used by external billing system. 

Couple[][]GetUsageRecordsForGroup (String groupName) 
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Application Management Component 
AddAppiication 

int AddApplication(String appname, String version, String description) 
Inputs: 

Appaname: Application name. 
Appversion: Application version 
Description. Application description. 
Outputs: 

-1 for failure to add the application. 
>0 otherwise. Application ID. 
Comments: 

Returns an app id for a newly added application. 

Errors: 

APPLICATION EXISTS 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetApplicationld 

int GetApplicationId(String appname, int version) 
Inputs: 

Appaname: Application name. 
Appversion: Application version 
Outputs: 

-1 for failure to find the application. 
X) otherwise. 
Comments: 

Returns an app id for a given application. 

Errors: 

NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 

GetApoIicationld 

int GetApplicationId(int Subscription^) 

Inputs: 

Subscription^! 
Outputs: 

0 for failure to find the application. 
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>0 otherwise. 
Comments: 

Returns an app id for a given application. 

Errors: 

NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 



GetSubscribedApplicationlds 

Int[]* GetSubscribedApplicationId(String username) 
Inputs: 

Usemame: username. 
Outputs: 

Array of application ids subscribed by a user. 
Comments: 

Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetUnsubscribedApplicationlds 
Int[]* GetUnsubscribedApplicationId(String username) 
Inputs: 

Username: username. 
Outputs: 

Array of application ids not subscribed by a user. 
Comments: 

Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



Couple[] GetApplicationDetail(int appid) 

Inputs: 

Application Id. 
Outputs: 
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Array of couple for the app id containing: 

{appname, appversion, description} values. 

Comments: 
Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



Component design 

We will discuss some complex scenarios in this section. 
Subscription 

Single New User 

1 . Create the user. CreateUser. 

a. If a user already exists, return error message and go back to 1 . 

2. Create the account for the user. CreateAccount 

a. Get the contact information from the user. 

b. Prompt to get the billing information. The user may decide to not give the 
billing information at this point. 

Corporate group admin creating an account 

1 . Create the admin user. CreateUser. 

2. Create the group. CreateGroup 

3. Create the account information for the group. CreateAccount. 

c. Get the contact information from the user. 

d. Prompt to get the billing information. 

4. Add users to the group. AddUserToGroup. 

a. This method will automatically create the user if they do not already exist 
in the system. 

b. The list of users is accessible to the Group Admin by querying: 

i. Our database GetUserRecords OR 



Jlj. 



Single User subscribing to an application 
1 . Validate the user. ValidateUser 
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2. Prompt to get the billing information if the billing information is not already pre- 
sent. 

3. Get the list of un-subscribed applications. GetUnsubscribedApplications. 

a. GetUnsubscribedApplicationlds. 

b. For each app id returned, get the application details. GetApplicationDe- 
tail 

4. For each additional application user wants to subscribe, call AddSubscription 
SUM server checking out an access token to use an application 

1. Call DBAcquireAccessToken. 



Testing design 

This document must have a discussion of how the component is to be tested. Some sub- 
sections could include: 

Unit testing plans 
Stress testing plans 
Coverage testing plans 
Cross-component testing plans 

Upgrading/Supportability/Deployment design 

This document must have a discussion of how the component addresses any specific is- 
sues related to upgrading, supporting and deployment of e-stream applications. Some ex- 
amples include: error conditions detected and reported by this component, any special 
hooks this component will provide for monitoring, hints for troubleshooting problems, 
any special hooks for debugging this component. 

Open Issues 

This is a list of issues that need to be further investigated or revisited during implementa- 
tion. 
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The following proposal is based on the C++ coding standards document available at 
http : / /www . possibility . com/Cpp/CppCodingStandard . html . This document will 
concisely present the coding standards from the coding standard document. The reader 
should refer to the original document (linked above) for a detailed explanation of the 
standards. This document has the following sections: 

NAMES: This section contains the naming schemes. 

ERRORS AND ERROR CODES: This section contains the error formats and the error 
codes to be used by eStream 1 .0 client and server components. 
FORMATTING: Code layout and formatting guidelines. 
COMMENTS: Guidelines for applying comments to the code. 
LOOSE END: Loose ends. 

Table of Contents: 



NAMES 2 

ERRORS AND ERROR CODES: 4 

FORMATTING 4 

COMMENTS 6 

LOOSE ENDS: 8 
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ID 


RULE 


Variables 


• Use upper case as word separators, lowercase for the rest of the word. 

• No underbars ('_/) 

• First letter of the variable could be upper/lower case. 
Examples: 

short Status; 

unsigned long timeOfDay; 


Pointers 


• Prepend the variable with p. 
Examples: 
string* pName; 
char** ppValue; 


Class Names 


• Use upper case letters as word separators, lower case for the rest of a word 

• First character in a name is upper case 

• For externally exposed components, use the first 3 words to denote the 
component. 

• No underbars ('_') 
Examples: 

• class ConfigurationManager 

• class Config 


Library Class 
Names 


• Prefix the elassname with OT 
Examples: 

• class OTHttpListener 

• class OTServer 


Class Method 
Names 


Same rule as for class names except for interfaces where the rule is: 

• Prefix the interface with the component's name. 

• Method names may optionally start with a lower case letter. 
Examples: 

• ECMGetFileld 

• Moni torGet ServerS et 

• monitorlnitialize. 


Class Attribute 
Names 


• Attribute names should be prepended with the character *m\ 

• After the W use the same rules as for class names. 

• W always precedes other name modifiers like 'p* for pointer. 

mc inii.cn; 
char* mpName; 
string* mpValue; 


Global 
Variables 


• Global variables should be prepended with "g'\ 
Examples: 
int gFlag; 
Logger gLog; 
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Logger* gpLog; 


Global 
Constants 


• Global constants should be all caps with 'J separators. 
Examples: 

const int A_GLOBAL_CONSTANT= 5; 


#defines and 
Macros 


• Put #defines and macros in all upper using 9 J separators 

Examples: 

#define MAX(a,b) blah 

#define IS_ERR(err) blah 


Function 
Names 


• Use upper case letters as word separators, lower case for the rest of a word 

• First character in a name is upper case 

• No underbars ( r J) 
Examples: 

• int SomeBloodyFunction() 


Enum Names 


• all upper using 9 J separators 

Examples: 

enum PinStateType 

{ 

PIN_OFF, 
PIN_ON 

} 




File Names 


• File should be all lower case 

• File name format should be <component>_<sub-component>.* 
Examples: 

monitor_heartbeat.cpp 
coreconfi gmanager.c 
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ERRORS AND ERROR CODES 



The following pound defines should be used for returning all successes and failures. 

#define SUCCESSWITHINFO -1 
#define SUCCESS 0 

#define FAILURE X) (The number representing an Error ID). 

All error messages will be prepended with an error code. The format for error code will 
be as follows: 

[ERRORID] [Severity] [Error Message] 
where, 

1 . ERRORID are unique across the system. 

2. Severity can be one of the following: 

a. 1-Low : A warning which can be ignored. 

b. 2-Medium: A warning which needs to be looked into. 

c. 3-High: Recoverable error in the component. 

d. 4-Critical: Irrecoverable error. Needs admin assistance. 

3. The error message in itself should have the following format: 
[COMPONENT] : [ERROR MESSAGE] :[ WORK AROUND] 

Error Ids distribution for client and server are as follows: 

0-1000 Server Internal Error Codes. 
1001 - 8000 Server Error Codes. 
800 1 - 9000 Client Internal Error Codes. 
9001-16000 Client Error Codes. 

FORMATTING 

The following formatting policies should be followed by all code. 
Braces Policy. 



if(0 = a) 
i 

} 

else 
{ 
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Indentation/Tabs/Space Policy 

Use the standard Visual C++ settings which are(using Tools->Options->Tabs menu): 

Indent Size: 4 

Auto Indent: Smart 

100 previous lines used for context. 

White spaces should be spaces and NOT tabs. 

VC++: Select "Insert Spaces" option. (This is NOT the default). 

Emacs: Refer http://www.delorie.com/gnu/docs/emacs/emacs_205.html 

Line Size etc. 

1 . Line size should not exceed 78 characters. 

2. There should be one statement per line. The following piece of code violates this 
principle. 

if (a>b) a++; 

Method/Functions Formats. 

1 . Methods should preferably be less 50 lines of code. 

2. Methods should not have more than 4 levels of nesting. 

3. Methods should preferably be re-entrant. Non-reentrant methods should be clearly 
marked as such. 

4. Each method/function should be preceded with a comment describing the method: 
/*******************^^ 

FUNCTION: 

INPUTS: 

OUTPUTS: 

DESCRIPTION: 

ERRORS: 

*******************^^ 
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COMMENTS 

Every file should start with the following Copyright disclaimer: 
* The Omnishift Software License, Version 1 .0 




mishift Technology. All rights 



* 



* Redistribution and use in source and binary forms, with or without 

* modification, are not permitted 
* 

*/ 
L 

2. Every decision should have comments. The following keyword are associated with 
decisions: 

a. if, else 

b. while, continue 

c. switch, case, default, break 

d. goto 

e. return 

3. Every class should have comment header with the following format: 

CLASS NAME: 
DESCRIPTION: 
FRIEND CLASSES: 
INCLUDES: 
LIBRARIES: 

************************************************* ******************/ 

4. Every function/method should have a header, (described above). 

5. Every file should have a header describing the contents of the file. 

6. Every directory should have a README describing the contents of the directory. 

7. Make GOTCHAS explicit. Use the following format for gotchas. 

• :TODO: topic <Author>:<Date> 
Means there's more to do here, don't forget. 

• :BUG: [bugid] topic <Author>:<Date> 

means there's a Known bue here, explain it and ontionallv give a hue ID. 

When you've done something ugly say so and explain how you would do it 
differently next time if you had more time. 

• :TRICKY: <Author>:<Date> 

Tells somebody that the following code is very tricky so don't go changing it 
without thinking. 
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rWARNING: <Author>:<Date> 
Beware of something. 
:COMPILER: <Author>:<Date> 

Sometimes you need to work around a compiler problem. Document it. The 

problem may go away eventually. 

: ATTRIBUTE: value <Author>:<Date> 

The general form of an attribute embedded in a comment. You can make up your 
own attributes and they'll be extracted. 



where an example of the AuthorrDate format would be: Bhaven Avalani J 
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LOOSE ENDS 



The following section notes some loose ends which do not fall in any of the categories 
above: 

1 . Always initialize all variables every time. 

2. Use header file guards against multiple inclusions of the header file. The guards 
would look like: 

#ifiidef ClassNameJi 
#define ClassName_h 

#endif // ClassName_h 

3. Object constructors should just initialize data. (They cannot return errors). Explicit 
Initialize() ca Hs should be made to do any involved work. 

4. Use continue and goto sparingly. 

5. Be "const" correct. Use "const" wherever and whenever applicable. 

6. All classes must have a Default Constructor and a Copy Constructor 

7. Set the compilation flag "Warnings as error" in Project -> Settings -> C/C++. 
This will show all warnings as errors. 

8. Use std: : namespace for all STL classes. 
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//develop/eng/docs/readme . txt 

This directory is for keeping all engineering related documents, 
but not product specific documents. Product specific documents are 
to be created with the product <docs> di rectory . 
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Software Development Process 



Version 0.2 
Ricky Benitez 



We are what we repeatedly do, 
Excellence, then, is not an act, but a habit 

ARISTOTLE 



This document describes the software development process followed at OTI for all 
software products. The purpose of this process is to ensure that software products are 
developed in an effective, predictable, repeatable and continuously improvable manner. 
The process has inputs, methods, outputs and metrics to determine its effectiveness. 

Ownership 

The overall responsibility for the development, refinement, effectiveness and adherence 
to the software development process belongs to the VP of Engineering. Responsibility for 
the various steps in the process will be assigned to appropriate individuals depending on 
the scope of the product and the makeup of the development group. 

Inputs 

The inputs to the software development process are: 

1 . Marketing Requirements Document (MRD) 

2. Technology White Papers and Prototypes 

Marketing Requirements Document 

The VP of Marketing is responsible for the development of this document. The VP of 
Engineering is responsible for extracting the portions of the MRD that will be 
implemented in software and capturing these as the High Level Requirements document. 

Technology White Papers 

The Chief Technology Officer is responsible for providing any technology white papers 
to convert the marketing requirements into a software product. 

Methods 

The primary methods used during the development process are described in detail later in 
the document, but primarily consist of: 

1 . Transforming the MRD to the High Level Requirements document (HLR) 

2. Transforming the HLR to the Product Datasheet (PDS) 
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3. Transforming the HLR to the High Level Design document (HLD) 

4. Transforming the HLR to the Product Test Plan (PTP) 

5. Transforming the HLD to the Low Level Design documents (LLDs) 

6. Transforming the HLD to an End-to-end Test Infrastructure (ETI) 

7. Transforming the LLDs to an Integration Test Infrastructure (ITI) 

8. Transforming the LLDs to Product Source Code (PSC) 

9. Integration 

10. Final Validation 

11. Performing a Post Mortem 

Outputs 

The outputs of the software development process are: 
L Updated MRD 

2. Product Datasheet (PDS) 

3. Digital components that meet the requirements listed in the HLR 

4. A set of mutually consistent and appropriately labeled design documents, source 
code, build environment, build infrastructure, test plans and test harnesses in a 
revision control system 

5. An issue database which contains a description of all past and outstanding issues 
relating to the requirements, design and implementation of the software 
components of the product 

6. A post mortem report indicating what was learned during the development 
process that can be used to improve the process in the future. 

Metrics 

Metrics are needed to track the progress of the development process and to constantly 
improve and fine-tune it. The primary mechanisms employed to measure are: 

1 . Completion of the methods 

2. The product issue database 

3. The process issue database 

These will be discussed in detail later in this document. 

Overview of the Process 

The software development process is iterative. While it may be described in a linear 
fashion, it must be understood that events will often dictate that previously completed 
portions of the process need alterations and that these alterations may propagate forward 
or backward along the various stages of the process. The ultimate objective is to end up 
with a set of outputs that are consistent relative to each other and relative to the inputs. 

Supporting Documents 

The following documents exist in conjunction with this one to support and fully define 
the overall software development process: 

1 . Coding Guidelines 

2. Configuration and Release Management Guidelines 
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Description of the Methods 



MRD to HLR 

The input of this method is a marketing requirements document and its output is a high- 
level requirements document. This method is performed once on a static MRD and then 
iterates incrementally as events warrant. Every change to the MRD after the initial HLR 
is produced generates an issue. The HLR must be kept in synch with the MRD and vice- 
versa. Once the initial HLR is produced changes are made only as a step towards 
resolving an issue. 

Description 

The HLR is a set of precise imperatives that collectively define the scope and behavior of 
the software product. Each precise imperative is known as a requirement. Requirements 
are grouped into the following categories: 



1. 


Functionality 


2. 


Localization 


3. 


Usability 


4. 


Reliability 


5. 


Performance 


6. 


Scalability 


7. 


Security 


8. 


Portability 


9. 


Maintainability 



Every product will have at least one requirement within each of these categories. Each 
requirement consists of the following: 

1 . Unique number - used to identify the requirement in other documents 

2. Description - a concise description of the requirement 

3. Importance - the level of importance to the final product between 1 andl 0. An 
importance of 1 indicates that the failure to fully or partially meet that 
requirement will have very minimal impact on the success of the product. An 
importance of 10 indicates that the product must either be able to meet the 
specified requirement or should not be produced at all 

Use cases are also included in an HLR to describe the various scenarios that the product 
is expected to handle from the perspective of its various users. Use cases are presented 
with the following information: 

1 . Summary - a description of the use or activity 

2. Actors - a listing of those who have a role in the activity 

3. Inputs - the initial state of the product and additional data that the actors need to 

4. Processing - the sequence of steps taken by the actors 

5. Output - the final state of the system and any data that the system produces 

Template 

An HLR template is located in the OTI share under \\fservl\oti\general\templates. 
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Review 



Upon completion of the HLR, a requirements review will be held. The requirements 
review process is as follows: 

1 . No less than four primary reviewers are selected, one should be from the 
engineering team (preferably the architect of the system), one should be from 
product marketing, one should be from the release group and one should be from 
the deployment or PSO organization. 

2. The primary reviewers are given a copy of the MDR and the HLR document 
sufficiently prior to the review meeting to review the HLR. 

3. The primary reviewers ensure to the best of their ability that: 

• The requirements are precise 

• There are appropriate requirements for every category (or a good 
understanding as to why no requirement is needed for a particular 
category) 

• The requirements and their importance level will, if faithfully transformed 
into a software product and incorporated with appropriate marketing, 
deployment and support, result in a successful customer solution 

4. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primary reviewers are 
present and have reviewed the requirements. 

5. If the requirements have not been fully reviewed by the primary reviewers, the 
review must be rescheduled. 

6. Primary reviewers bring up their issues and a decision is made as to whether the 
item is or is not a real issue. The scribe logs all real issues. 

7. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 

8. The scribe then polls the primary reviewers to determine which of the following 
courses of action will be taken: 

• HLR is completed after all issues are satisfactorily resolved 

• HLR must be reviewed again after all issues are satisfactorily resolved 

• HLR must be abandoned and a new requirements effort begun 

9. After the meeting, the scribe is responsible for filing and submitting the review 
report. If the HLR is not to be abandoned, the scribe is also responsible for 
entering all issues into the issue tracking system. 

Completion 

The MDR to HLR method is considered complete only after: 

• a review of the HLR whose primary reviewers decide should be considered 




• 



• 



a review report has been submitted for every review of the HLR 

all HLR issues have been submitted and satisfactorily resolved 

the HLR has been appropriately added to the revision control system 
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Issue Resolution 

HLR issues that result in the removal, change or inclusion of one or more requirements 
can be considered resolved after the HLR and any other document that are affected and 
must be kept in synch with the HLR are appropriately updated and each change has been 
reviewed and approved by one reviewer from the engineering team, one from the 
marketing team, one from the product marketing group, one from the release group and 
one from the deployment or PSO group. If the HLR is significantly changed as a result of 
addressing an issue, a full review should be considered. 



HLR to PDS 

The input to this method is a high-level requirements document and the output is a 
product datasheet. This method is performed once on the HLR and then iterates 
incrementally as events warrant. The PDS must be kept in synch with the HLR and vice- 
versa. Once the initial PDS is produced changes are primarily made only as a step 
towards resolving an issue. The speculative nature of the PDS gives its owner more 
leeway during review and editing than is accorded to the owners of most other product 
documents and deliverables. 



Description 

The PDS is a document consisting of various sections that describe the overall software 
product to potential stakeholders. The PDS contains a section for each of the following: 

1 . Is/Is Not - this section describes the product in terms of what it is and, to dispel 
potential misconceptions about what it might be, what it is not 

2. Flexibility Matrix - this section indicates the relative importance of time, features 
and cost 

3. Requirements - this section is a copy of the HLR, potentially tailored to a less 
technical audience 

4. Schedule - this section lists the major milestones of the product 

5. Dependencies - this section lists the major external dependencies of the product 

6. Resources - this section lists the resources needed by engineering to complete the 
product as follows: 

a. Engineers and managers needed per month 

b. Machine resources needed 

c. Software resources (licenses, etc.) needed 

7. Quality Plan - this section lists the steps that will be taken towards reaching the 
quality goals of the product: 

a. What will be minimally prototyped prior to high-level design 

b. Any exceptions or additions to the software development process defined 

• . ».t r ....... .„ ... u 

c. Coverage, number of white-box tests and other criteria required to meet 
completion on PSC and final validation 

d. Alpha, beta and other pre-production releases 

8. Risks - this section lists the anticipated risks and appropriate contingency plans to 
address those risks. 
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Template 

A PDS template is located in the OTI share under \\fservl\oti\general\templates. 
Review 

Upon completion of the PDS, it is posted for the managers, engineers and stakeholders of 
the product to review. 

Completion 

The HLR to PDS method is considered complete only after: 

• The PDS has been completely filled to the best understanding of its owner 

• The PDS has been posted for review 

Issue Resolution 

PDS issues that result in changes to the PDS are considered resolved after the PDS and 
any other document affected that must be kept in synch with the PDS are appropriately 
updated and reviewed. 

HLR to HLD 

The input of this method is a high-level requirements document and the output is a high- 
level design document. This method is performed once on the HLR and then iterates 
incrementally as events warrant. The HLD must be kept in synch with the HLR and vice- 
versa. Once the initial HLD is produced changes are made only as a step towards 
resolving an issue. 

Description 

The HLD is a document consisting of various sections that describe the overall software 
product in high-level form. The HLD contains a section for each of the following: 

1. Definition of terms - this section defines each technical term that may lead to 
confusion in the understanding of the design if left undefined 

2. Block diagram - a pictorial description of the system, to aid in the identification 
and understanding of the components and APIs described in the design 

3. High-level description of each component - a description of each major 
component of the system which clearly describes the role that each component 
plays in the overall system and serves as the launching point of the component's 
low level design 

4. High-level description of each API - a catalog and simple description of each 
major API in the system, which a particular emphasis on the system's external 

5. High-level test strategy ^ a description of the approach to be taken to validating 
the correctness of the system end-to-end 

Template 

An HLD template is located in the OTI share under \\fservl\oti\general\templates. 
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Review 

Upon completion of the HLD, a low-level design review will be held. The HLD review 
process is as follows: 

1 . No less than two primary reviewers are selected. 

2. The primary reviewers are given a copy of the HLR and the final HLD document 
sufficiently prior to the review meeting to review the HLD. 

3. The primary reviewers ensure to the best of their ability that: 

• The design meets its requirements 

• All sections of the design are complete 

• The design is as simple, robust, testable, scalable and well thought through 
as time requirements permits 

• Competent designers could perform a low-level design of each component 
given the HLR and HLD without needing to consult the author of the HLD 

4. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primary reviewers are 
present and have reviewed the design. 

5. If the design has not been fully reviewed by the primary reviewers, the review 
must be rescheduled. 

6. Primary reviewers bring up their issues and a decision is made as to whether the 
item is or is not a real issue. The scribe logs all real issues. 

7. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 

8. The scribe then polls the primary reviewers to determine which of the following 
courses of action will be taken: 

• Design is completed after all issues are satisfactorily resolved 

• Design must be reviewed again after all issues are satisfactorily resolved 

• Design must be abandoned and a new design effort begun 

9. After the meeting, the scribe is responsible for filing and submitting the review 
report. If the design is not to be abandoned, the scribe is also responsible for 
entering all issues into the issue tracking system. 

Completion 

The HLR to HLD method is complete only after: 

• a review of the HLD whose primary reviewers decide should be considered 
completed after all issues are resolved has transpired 

• a review report has been submitted for every review of the HLD 

• all HLD issues have been submitted and satisfactorily resolved 

o o*iv r>hori croc rf*fniTf*A *r> tJ>~ TJT I? try J>rir»r? Ko*V| J^J P on^ TJT As\r+~-~~r>ni r 

s^ii^ii nave oecu buonintcu as i^ucs 

• the HLD has been appropriately added to the revision control system 

Issue Resolution 

HLD issues that require changes to the HLD can be considered resolved after the HLD 
document and any other documents that are affected and must be kept in synch with the 
HLD are appropriately updated and each change has been reviewed and approved by at 
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least one reviewer. If the HLD is significantly changed as a result of addressing an issue, 
a full review should be considered. 



HLR to PTP 

The input to this method is the high-level requirements document and the output is a 
product test plan. This method is performed once on a static HLR and iterates 
incrementally as events warrant. The PTP must be kept in synch with the HLR and vide- 
versa. Once an initial PTP is produced changes are made only as a step towards resolving 
an issue. 

Description 

The PTP is a document describing the various black-box and stress tests that will be 
applied to the product to ensure that it meets its stated requirements. The PTP should 
completely cover every requirement in the HLR to ensure that the delivered product 
meets its stated goals. 

Template 
TBD 

Review 

Upon completion of the PTP, a test plan review will be held. The PTP review process is 
as follows: 

1 . No less than two primary reviewers are selected. 

2. The primary reviewers are given a copy of the final PTP document sufficiently 
prior to the review meeting to review the PTP. 

3. The primary reviewers ensure to the best of their ability that: 

• There are appropriate tests defined to cover all the requirements 

• The test are as simple, robust, repeatable, scalable and well thought 
through as time requirements permits 

• A competent SQA engineer could conduct the test using the PTP without 
needing to consult its author 

• The PTP meets the quality criteria stated in the quality plan section of the 
PDS 

4. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primary reviewers are 
present and have reviewed the test plan. 

5. If the test plan has not been fully reviewed by the primary reviewers, the review 
must be rescheduled. 

item is or is not a real issue. The scribe logs all real issues. 

7. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 

8. The scribe then polls the primary reviewers to determine which of the following 
courses of action will be taken: 

• Plan is completed after all issues are satisfactorily resolved 
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• Plan must be reviewed again after all issues are satisfactorily resolved 

• Plan must be abandoned and a new planning effort begun 

9. After the meeting, the scribe is responsible for filing and submitting the review 
report. If the plan is not to be abandoned, the scribe is also responsible for 
entering all issues into the issue tracking system. 

Completion 

The HLR to PTP method is complete only after: 

• a review of the PTP whose primary reviewers decide should be considered 
completed after all issues are resolved has transpired 

• a review report has been submitted for every review of the PTP 

• all PTP issues have been submitted and satisfactorily resolved 

• any changes required to the HLR to bring this documents in synch with the PTP 
have been submitted as issues 

• the PTP has been appropriately added to the revision control system 

Issue Resolution 

PTP issues that require changes to the PTP can be considered resolved after the PTP 
document and any other documents that are affected and must be kept in synch with the 
PTP are appropriately updated and each change has been reviewed and approved by at 
least one reviewer. If the PTP is significantly changed as a result of addressing an issue, a 
full review should be considered. 

HLD to LLDs 

The input to this method is a high-level design document and the output is one low-level 
design document for each component described in the HLD. This method is performed 
once on the HLD and then iterates incrementally as events warrant. The LLD must be 
kept in synch with the HLD and vice-versa. Once the initial LLD is produced changes are 
made only as a step towards resolving an issue. 

Description 

The LLD is a document consisting of various sections to describe the low-level design of 
a component. The LLD contains a section for each of the following: 

1 . Functionality - this section describes the functionality that this component 
provides 

2. Data type definitions - a list of the data types defined or used by this component 

3. Data structures - a description of the data structures defined and used by this 

component 

supported by this component — for user interfaces, a mock-up or prototype of the 
interface must be provided (separate from the LLD) and appropriate screen shots 
of this prototype should be included 
5. Component design description - a description of the component sufficient to 
allow a competent programmer to implement the component without needing to 
consult the original author 
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6. Testing - a description of the test harnesses used to unit test, stress test and/or 
coverage test this component sufficient to allow a competent programmer to 
implement these harnesses without consulting the author 

7. Supportability - a description of the supportability features of the component such 
that an integration engineer can make full use of the supportability features built 
into the component without needing to consult the author or developer 

Template 

An LLD template is located in the OTI share under \\fservl\oti\general\templates. 
Review 

Upon completion of the LLD, a low-level design review will be held. The LLD review 
process is as follows: 

1 . No less than two primary reviewers are selected. 

2. The primary reviewers are given a copy of the final LLD document sufficiently 
prior to the review meeting to review the LLD. 

3. The primary reviewers ensure to the best of their ability that: 

• The design meets its requirements 

• All sections of the design are complete 

• The design is as simple, robust, testable, scalable and well thought through 
as time requirements permits 

• A competent software engineer could implement the component using the 
LLD without needing to consult its author 

4. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primary reviewers are 
present and have reviewed the design. 

5. If the design has not been fully reviewed by the primary reviewers, the review 
must be rescheduled. 

6. Primary reviewers bring up their issues and a decision is made as to whether the 
item is or is not a real issue. The scribe logs all real issues. 

7. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 

8. The scribe then polls the primary reviewers to determine which of the following 
courses of action will be taken: 

• Design is completed after all issues are satisfactorily resolved 

• Desigji must be reviewed again after all issues are satisfactorily resolved 

• Design must be abandoned and a new design effort begun 

9. After the meeting, the scribe is responsible for filing and submitting the review 

entering all issues into the issue tracking system. 
Completion 

The HLD to LLD method is complete only after: 

• a review of the LLD whose primary reviewers decide should be considered 
completed after all issues are resolved has transpired 
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• a review report has been submitted for every review of the LLD 

• all LLD issues have been submitted and satisfactorily resolved 

• any changes required to the HLD to bring this document in synch with the LLD 
have been submitted as issues 

• the LLD has been appropriately added to the revision control system 

Issue Resolution 

LLD issues that require changes to the LLD can be considered resolved after the LLD 
document and any other documents that are affected and must be kept in synch with the 
LLD are appropriately updated and each change has been reviewed and approved by at 
least one reviewer. If the LLD is significantly changed as a result of addressing an issue, 
a full review should be considered. 



HLD to ETI 

The input to this method is a high-level design document with a high-level test strategy 
description and the output is an end-to-end test infrastructure. This method is performed 
once on the HLD and then iterates incrementally as events warrant. The ETI 
infrastructure must be kept in synch with the high-level test strategy in the HLD and vice- 
versa. Once an initial ETI is produced changes are made only as a step towards resolving 
an issue. 

Description 

The ETI is a set of scripts, source files, include files, make files and documents that 
collectively are used to automate the process of performing a rigorous end-to-end test of 
the integrated PSC. The purpose of the ETI is to provide an automated method for 
ensuring that the overall integrated PSC operates on a regular (nightly) basis. 

Template 
TBD 



Review 

Upon completion of the ETI, a code review will be held. The code review process is as 
follows: 

1 . No less than two primary reviewers are selected. 

2. The primary reviewers are given a copy of the ETI and related code (or pointed to 
their location in the configuration management system) sufficiently prior to the 
review meeting to review the code. 

3 . The orimarv reviewers ensure to the best of their ».W1i*v tViot- 

* a iic coae is an eiiieiem, i easonaoie ana complete implementation oi the 
high-level test strategy. Efficiency means that the end-to-end test does not 
require manual steps and can complete in no more than a couple of hours. 

• A competent SQA engineer could maintain the ETI without needing to 
consult its author (assuming that the HLD is also available) 
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4. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primary reviewers are 
present and have reviewed the implementation. 

5. If the implementation has not been fully reviewed by the primary reviewers, the 
review must be rescheduled. 

6. Primary reviewers bring up their issues and a decision is made as to whether the 
item is or is not a real issue. The scribe logs all real issues. 

7. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 

8. The scribe then polls the primary reviewers to determine which of the following 
courses of action will be taken: 

• Implementation is completed after all issues are satisfactorily resolved 

• Implementation must be reviewed again after all issues are satisfactorily 
resolved 

• Implementation must be abandoned and a new implementation effort 
begun 

After the meeting, the scribe is responsible for filing and submitting the review report. If 
the implementation is not to be abandoned, the scribe is also responsible for entering all 
issues into the issue tracking system. 

Completion 

The HLD to ETI method is complete only after: 

• a review of the ETI whose primary reviewers decide should be considered 
completed after all issues are resolved has transpired 

• a review report has been submitted for every review of the ETI 

• all ETI issues have been submitted and satisfactorily resolved 

• any changes required to the HLD test strategy to bring this documents in synch 
with the ETI have been submitted as issues 

• the ETI has been appropriately added to the revision control system as specified 
in the configuration and release management guidelines 

• the ETI has been set up to run automatically on each successful and sanity-tested 
build of the integrated PSC and the results of the test are being mailed to all 
interested parties 

Issue Resolution 

ETI issues that require changes to the test infrastructure must be appropriately reviewed 
before they are closed. The review process depends on the nature of the changes made to 
the ETI. If only one to twenty-five source statements are affected, then a desk check 



member needs to add their review comments to the issue prior to closing it. If more than 
twenty-five statements are added or modified, then the standard review process for ETI 
must be followed. 
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LLDtolTI 

The input of this method is the set of low-level design documents with component test 
strategies and the output is an integration test suite definition and infrastructure. This 
method is performed once after all or after a substantial portion of the LLDs have been 
produced and then iterates incrementally as events warrant. The 1TI must be kept in 
synch with the LLDs and vice-versa. Once the initial ITI is produced changes are made 
only as a step towards resolving an issue. 

Description 

The ITI is a set of scripts, source files, include files, make files, test definition files and 
documents that collectively are used to automate the process of performing a rigorous set 
of regression, black-box, stress, coverage, performance and sanity tests on the PSC. The 
purpose of the ITI is to provide an automated method for ensuring that the individual 
product components and the overall integrated PSC is robust on a regular (nightly) basis. 

Template 
TBD 

Review 

Upon completion of the ITI, a test suite and code review will be held. The code review 
process is as follows: 

1 . No less than two primary reviewers are selected. 

2. The primary reviewers are given a copy of the test suite and related code (or 
pointed to their location in the configuration management system) sufficiently 
prior to the review meeting to review the suite and the code. 

3. The primary reviewers ensure to the best of their ability that: 

• The test suite will efficiently provide a high-level of confidence that every 
component of the PSC is of high quality. Efficiency means that the entire 
ITI runs in no more than a few hours. This entails examining each test 
suite and ensuring that tests are not overly redundant or spend inordinate 
amounts of time running while providing only marginal improvements in 
the confidence level of the product's quality. 

• The code is reasonable and complete automation of the integration test 
suite 

• A competent SQA engineer could maintain the ITI without needing to 
consult its author (assuming that the LLDs are also available) 

4. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primarv reviewers are 

5. If the implementation has not been fully reviewed by the primary reviewers, the 
review must be rescheduled. 

6. Primary reviewers bring up their issues and a decision is made as to whether the 
item is or is not a real issue. The scribe logs all real issues. 

7. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 
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8. The scribe then polls the primary reviewers to determine which of the following 
courses of action will be taken: 

• Test suite and implementation is completed after all issues are 
satisfactorily resolved 

• Test suite and implementation must be reviewed again after all issues are 
satisfactori ly resolved 

• Test suite and implementation must be abandoned and a new 
implementation effort begun 

After the meeting, the scribe is responsible for filing and submitting the review report. If 
the test suite and implementation is not to be abandoned, the scribe is also responsible for 
entering all issues into the issue tracking system. 

Completion 

The HLD to ITI method is complete only after: 

• a review of the ITI whose primary reviewers decide should be considered 
completed after all issues are resolved has transpired 

• a review report has been submitted for every review of the ITI 

• all ITI issues have been submitted and satisfactorily resolved 

• any changes required to the LLD test strategies to bring this documents in synch 
with the ITI have been submitted as issues 

• the ITI has been appropriately added to the revision control system as specified in 
the configuration and release management guidelines 

• the ITI has been set up to run automatically after each successful build of the 
integrated PSC and the results of the test are being mailed to all interested parties 

Issue Resolution 

ITI issues that require changes to the test suite or the source code must be appropriately 
reviewed before they are closed. The review process depends on the nature of the 
changes made to the ITI. If changes are one to several additional tests added to an 
existing test suite, then a desk check of each added test by another member of the 
technical staff is all that is required. If a new test suite or more than several new test cases 
are being added, then the standard review process for the ITI must be followed. If 
changes are to source code and only one to twenty-five source statements are affected, 
then a desk check review performed by another member of the technical staff is all that is 
required. That member needs to add their review comments to the issue prior to closing 
it. If more than twenty- five statements are added or modified, then the standard review 
process for ITI must be followed. 



The input to this method is a low-level design document and the output is product source 
code. This method is performed once for each completed LLD and then iterates 
incrementally as events warrant. The LLD must be kept in synch with the PSC and vice- 
versa. Once the initial PSC is produced changes are made only as a step towards 
resolving an issue. 
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Description 

The PSC is a collection of the source files, include files, make files and a collection of 
development tools and an environment that will be used to produce the binary bits that 
will be ultimately delivered to a customer. Additional supporting code including, but not 
limited to code generators, component tests, white box tests, regression tests and test 
drivers are produced while applying this method. PSC must adhere to the coding 
guidelines and the completion criteria for this method. PSC and all other supporting code 
must adhere to the configuration and release management guidelines. 

Template 

See the coding guidelines and configuration and release management guidelines for 
template information. 

Review 

Upon completion of the PSC, a code review will be held. The code review process is as 
follows: 

1 . No less than two primary reviewers are selected. 

2. The primary reviewers are given a copy of the PSC and related code (or pointed 
to their location in the configuration management system) sufficiently prior to the 
review meeting to review the code. 

3. The primary reviewers ensure to the best of their ability that: 

• The code is a reasonable and complete implementation of the LLD 

• The associated white-box and component tests specified in the LLD have 
been implemented and have passed successfully 

• The PSC adheres to the coding guidelines 

• A competent software engineer could maintain the PSC without needing to 
consult its author (assuming that the LLD and HLD are also available) 

4. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primary reviewers are 
present and have reviewed the implementation. 

5. If the implementation has not been fully reviewed by the primary reviewers, the 
review must be rescheduled. 

6. Primary reviewers bring up their issues and a decision is made as to whether the 
item is or is not a real issue. The scribe logs all real issues. 

7. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 

8. The scribe then polls the primary reviewers to determine which of the following 
courses of action will be taken: 

• Implementation must be reviewed again after all issues are satisfactorily 
resolved 

• Implementation must be abandoned and a new implementation effort 
begun 
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After the meeting, the scribe is responsible for filing and submitting the review report. If 
the implementation is not to be abandoned, the scribe is also responsible for entering all 
issues into the issue tracking system. 

Completion 

The LLD to PSC method is complete only after: 

• a review of the PSC and associated code whose primary reviewers decide should 
be considered completed after all issues are resolved has transpired 

• a review report has been submitted for every review of the PSC 

• all PSC and associated code issues have been submitted and satisfactorily 
resolved 

• any changes required to the LLD to bring this documents in synch with the PSC 
have been submitted as issues 

• the PSC and associated code has been appropriately added to the revision control 
system as specified in the configuration and release management guidelines 

• the PSC meets the coverage and component testing criteria specified in the quality 
plan section of the PDS 

Issue Resolution 

PSC issues that require changes to the source code must be appropriately reviewed before 
they are closed. The review process depends on the nature of the changes made to the 
PSC. If only one to twenty-five source statements are affected, then a desk check review 
performed by another member of the technical staff is all that is required. That member 
needs to add their review comments to the issue prior to closing it. If more than twenty- 
five statements are added or modified, then the standard review process for PSC must be 
followed. 

Integration 

The input to this method is the separate components product source codes, the integration 
test infrastructure and the end-to-end test infrastructure and the output is the fully 
integrated product source code. This method is performed iteratively starting as soon as 
there is more than one interacting component PSC available and as long as there are 
changes made to any of the component PSC. The method must take place as frequently as 
is practically possible while the PSC is changing. The desired goal is once per day. 

Description 

Integration is the process of bringing all of the PSC components together to produce the 
entire product and validating them against a suite of regression, black-box, stress. 

final validation. In order to provide developers with timely feedback concerning any 
integration and ETI failures that are introduced into the PSC, the integration process 
should be automated and performed as regularly as it is practical (every night, assuming 
that changes have been made to the PSC since the last successful integration). 
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Template 
TBD 

Review 

Integration is by definition reviewed every time it is attempted (about once a day). 
Completion 

Integration is complete and a candidate for final validate is produced when: 

• The entire system has been successfully built from the completed PSC of every 
component using only the documented build environment 

• The built system successfully passes all regression, black-box, stress, 
performance, sanity and ETI test suites designated for integration 

• All high and medium priority requirements, design and implementation issues 
have been resolved 

Issue Resolution 

There are rarely any issues against the integration phase. Most issues are ITI and ETI 
issues and are covered under those particular items. Issues might come up, however, in 
which integration is not taking place with the level of efficiency or producing the 
expected amount of confidence. In such cases, the issue may be categorized as an 
integration issue as a placeholder for determining whether it should be filed against ITI or 
ETI. 

Final validation 

The input to this method is a successful integration that produces a valid final validation 
candidate and the output is a fully validated release candidate. This method is applied as 
often as required, although it tends to be planned to coincide with code freeze and change 
control to ensure a quick and timely resolution of just those issues that are preventing the 
successful final validation of the product. 

Description 

Final validation is the process of taking a valid final validation candidate and applying 
the entire PTP against it. If every test defined in the PTP is successfully passed, then the 
candidate is declared to be a valid release candidate. Once this occurs, the build 
environment and all of the source files, include file, make files, test definition files and 
scripts that were used to produce the release candidate are appropriately labeled as stated 
in the configuration and release management guidelines. 

lemptate 
TBD 

Review 

Final validation is by definition reviewed every time it is attempted. 
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Completion 

The integration stage is complete and a candidate for release is produced when: 

• The entire PTP is successfully applied against a valid final validation candidate. 

• The build environment and all of the source files, include file, make files, test 
definition files and scripts that were used to produce the release candidate are 
appropriately labeled as stated in the configuration and release management 
guidelines. 

• No requirements, design, implementation or other types of issues logged against 
the current release with a priority greater than that stated as a requirement for 
release in the PDS remain unresolved. 

Issue Resolution 

There are rarely any issues against the final validation phase. Most issues are PTP issues 
and are covered under that category. 

Post Mortem 

The input to this method is a complete cycle through the software development process 
and the opinions of the participants. The post mortem method is applied once for each 
completed product release. The output is a post mortem report and a set of issues 
submitted against the software development process and any of the other supporting 
processes (such as the configuration and release management process). 

Description 

A post mortem captures the learning that took place over the software development 
lifecycle and should produce a number of issues against the software development 
process and other related processes that are to be resolved. If this is done soon after the 
completion of the development and release cycle and issues are addressed, then there is 
opportunity for continual refinement and improvement of the software development 
process and of the organization's development capability. The method requires that each 
participant fill out a feedback form and that these forms be reviewed by a set of 
individual who produce a post mortem report and file the issues that the reviewers 
determined were relevant against each appropriate process. 

Template 
TBD 

Review 

i lie icvicw piuCcSS ii> cl> iOllOWS: 

1 . No less than two primary reviewers are selected. 

2. All individuals who were asked to give feedback are invited to the review. 

3. The primary reviewers are given a copy of the feedback forms sufficiently prior to 
the review meeting to review the code. 

4. The primary reviewers ensure to the best of their ability that: 
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• AH feedback comments are considered and appropriate root causes of 
issues are identified 

• The issue tracking system is consulted to collaborate or disprove potential 
issues when appropriate 

5. At the start of the review, a scribe (not a primary reviewer) is selected. The scribe 
notes down who is in attendance and ensures that all primary reviewers are 
present and have reviewed the implementation. 

6. If the feedback has not been fully reviewed by the primary reviewers, the review 
must be rescheduled. 

7. Primary reviewers bring up their issues and a decision is made as to whether the 
item is or is not a real issue. The scribe logs all real issues. 

8. When the primary reviewers have presented all their issues, other attendees are 
invited to bring up issues. Real issues are logged. 

After the meeting, the scribe is responsible for filing and submitting the post mortem 
report and entering all issues into the issue tracking system. 

Completion 

The post mortem is not complete until: 

• Post mortem feedback has been received from all available participants 

• The post mortem review is held 

• All issues have been submitted against the appropriate process 

• A post mortem report has been submitted 

Issue Resolution 

All issue filed against the software development process after the post mortem must be 
resolved in a timely manner. All changes must be reviewed and approved by the VP of 
Engineering. 

Description of Metrics 

TBD 
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Estream 1.0 Planning Document 



Low-Level Design Status/Plan 
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Implementation Plan 
Milestones 




ECM (RAM disk cache) and EFSD executes a local "htmom" executable 

Photoshop is installed locally and successfully executed from estream sets and applnstallblk produced by builder 
App Server and EMS integrated to copy "himom" executable using a dummy client 
App Server, EMS and CNI integrated to copy "hi mom" executable from "hi mom" estream sets 
office is installed locally and successfully executed from estream sets and applnstallblk produced by builder 
App Server. EMS, ENI, ECM and EFSD integrated to run "himom" from estream sets on server 
Following applications built and tested with local installation 
Adobe Premier 

Lotus Suite 

Photoshop is installed by AIM and executed from estream sets on App server 

No Subscription 

No License Management 

RAM cache for ECM 

Installation of Photoshop using AIM 
Photoshop is installed by AIM and executed from estream sets on App server 

No Slim Server 

Disk based cache for ECM 



Estream includes initial prefetched pages and these pages are prefetched during installation 

Fully functional estream bits (includes initial prefetched pages) 

Client software is run as a service 

App Server is started by Monitor 

Admin Ul to stop and start app Server 

Application subscription from web server 

installation on client after subscription 
Testing environment is setup (configuration of 3 servers and one client) 
Photoshop runs with the following additional functionality 

Leads for milestone: Amit and Nick 

Slim Server 

http protocol 

CNI supports uique message ids for NAD 

Fully functional LSM 

Real Accesstokens 

Vninstall applications 

Anti-Piracy support 

AppServer and SlimServer faiU>ver 

File spoofing 



Clean builds by integration (George) (Raj will drive this) 

Office is running with full functionality 

Restruturing of client so it can be started at boot time 
Performance tuning 
Improve robustness 
application upgrade 
Crash resiliency 

All software purified and memory leaks eliminated 
(May be) Applets for monitoring server components 

Office is removed from desktop of at least one person and 
reinstalled using estream 



Code Freeze 



eStream 1.0 High Level Design 



Version 1.0 




Introduction 

This document describes the high level design for the eStream 1.0 product. It is 
essentially a summary and a tying together of the low level designs for each component 
in the system. The organization of this document is: 



□ Basic overview of the entire system 

□ Block diagrams for the client, server, and builder portions, showing all major 
components 

□ General discussion of each component, and pointers to the low level documents 
for these components 

□ A list of known issues 

To understand the problem being solved in this design, see the "eStream Requirements 
Document" for information. 



Note that this design is for a Windows NT4.0 and Windows 2000 client only As work 
progresses on a Windows 95/98 client, the designs here will be updated. 

% *■ «■ 

Overview 



eStream 1.0 encompasses the following basic features: 

1 . A distributed file system for application files, residing on a server and cached on a 
client. 

2. A small client "player" program to allow local execution of applications that 
reside on the servers. 

3. Authentication using tokens supplied by a license server to each active client. 

4. A managed database of information about applications available to client 
machines, and subscription and usage data for each registered user. 

5. Integration with service provider web servers to allow users to subscribe to apps 
and manage their accounts. 

' - ' • • r n , . i - - ... 

7. A build system that analyzes applications and enables them to be executed by the 
client and server. 

8. Anti-piracy features to discourage unauthorized copying and use of subscribed 
applications. 



As a way of overview, here are the processes that take place to enable and execute a 
Windows application from a client machine. 



eStream 1 .0 High Level Design 



version 0.3 



□ The eStream builder is used to create an eStream set for the application. The 
application is installed on a clean machine, with the builder tools running. These 
will monitor all file installs and registry updates required to run the application, 
and encode them into a binary file— the eStreamSet — that will be installed on a 
service provider's eStream application server (app server). 

□ A user must download and install the eStream client (ECE) onto her machine, and 
register as a valid user from a service provider; this will be done using the service 
provider's web site. 

□ The user will subscribe to an application from the service provider; a browser 
module on the client machine will be notified and send a message to the ECE 
about this event. 

□ The ECE will communicate with the service provider's eStream license server 
(Slim server) to verify the newly subscribed app and all permissions, and will 
install a small portion of the application onto the client system — essentially, the 
registry entries, shortcuts, and small shared files necessary for execution. 

□ All application files that are not installed on the client will be accessed via a 
separate eStream file system (EFSD) 

□ The user will now see standard shortcuts for subscribed applications, exactly as 
though the app were installed locally. 

□ Starting an application, via a command line or double-clicking a shortcut, will 
cause the client machine to start executing the application on the EFSD. This 
means the virtual memory manager will request pages from the EFSD during page 
faults. 

□ These requests will be forwarded from the EFSD to the eStream cache manager 
(ECM), a component of the ECE, and on to the app server, assuming the page 
requested is not in the cache. 

□ Before any page request is fulfilled by the ECE, the client license subscription 
manager (LSM) will check that the user has permission to run the application, 
requesting an access token from the Slim server if an existing one has expired. 

□ This valid access token is sent from the client to the app server for every page 
request; this authenticates the request. 

□ The server monitor will be continually checking the state of the app servers and 
Slim servers. If any are down, it will take them offline. 

a The client has a list of valid Slim and app servers for each registered service 
provider and subscribed application. If response time for any of these is bad, it 
will stop using it and fall back on the rest. 

Block diagrams 

The following are simple block diagrams of the client and server components. Some 
conventions: 

□ A box represents a logical eStream component. A component may exist as a 
distinct process, or it may be grouped with other components into a common 
process. 
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□ A line between components represents an interface call from one to another. If A 
calls B, there's a arrow on the end of the line at B. If A and B call each other, 
there's an arrow on both ends of the line. 

Note that data stores are not represented in these diagrams; if a data store is centrally 
managed, then there is a component that has interfaces to allow access to these data. 
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eStream Client Block Diagram 



SERVER 



ECE Components 
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eStream Server Block Diagram 




Generic 
Client 



Farm Manager 
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Component descriptions 
Client components 

The eStream client consists of the following components illustrated in the diagram above: 

□ ECE: the eStream Client Executable. This is the aggregation of several user space 
components into a single executable, operating as a Windows service. 

□ LSM: the License Subscription Manager (part of the ECE). This tracks and 
handles all required user information needed by the client: service providers, 
subscriptions, and access rights. 

□ AIM: the App Install Manager (part of the ECE). This is responsible for 
installing all necessary bits onto a client machine in order to run a subscribed 
application. It also uninstalls all local app bits when unsubscribing. 

□ ECM: the eStream cache manager (part of the ECE). This is the user-space 
component that handles requests from the EFSD, and manages the on-disk and in- 
memory cache of file contents. 

□ EPF: the eStream PreFetch component (part of the ECE). This works closely with 
the ECM to handle prefetches of pages for running eStream applications (as 
opposed to demand fetches, handled by the ECM). 

□ CNI: the Client Network Interface (part of the ECE). This manages queues of 
requests from various client components to the app and Slim servers. 

□ EMS: the eStream Message Service (part of the ECE). This library, used in both 
the client and servers, handles the actual network sends and receives between 
remote machines. 

□ CBM: the Client Browser Module. This is a client-side web browser plugin that 
is used to handle notification from a service provider's web server to the ECE, 
when user updates have taken place. 

□ CIN: the Client Installer module. This small component installs, upgrades, and 
uninstalls all the required client software. 

□ FSP: the File Spoofer. This is a kernel-mode driver that is used to redirect 
requests, intended for local filesystems, to the EFSD. It is a file system filter 
driver that sniffs all Create requests to the necessary local FSDs, compares the 
filenames with a list of files that must be spoofed, and if a match is seen, redirects 
the request to the EFSD. 

□ EFSD: the eStream File System Driver. This is a standard Windows NT FSD, 
handling all necessary FS requests from the I/O Manager. It ultimately sends 
these requests to the ECM to be satisfied (either locally or remotely). 

clustering for threads running as part of eStreamed applications. 
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ECE 



The ECE is the Windows service that comprises the bulk of the user-space eStream client 
software. It provides an overall main program loop, as well as the user interface 
component for all client components that must communicate with a user. 



The LSM tracks current subscription information and determines the need for license 
validation. It is informed of subscription changes from the client UI, and is queried by the 
ECM to validate accessibility to different applications, based on the license model for the 
subscription to that application. 

The LSM has a few major tasks: 

1 . Keep track of what subscriptions the current user has available from all ASPs 

2. Determine which application a given file is a part of 

3. Acquire an access token to validate a license for file requests that require one 
There are two ways that the LSM updates its list of known subscribed applications: 

1 . It may be informed of new subscriptions, or of applications that are unsubscribed, 
by the client UI, as part of a browser plugin in conjunction with an ASPs web site. 

2. It may asynchronously poll an ASPs Slim servers to get updated lists of 
subscribed apps. 



The AIM is the contact point for installation and uninstallation of applications on a client 
machine. It gets the requests from the LSM to install applications when the user 
subscribes to them, and it gets requests from the Client UI to uninstall applications. 

The AIM manages application installs on the client machine. It keeps track of what 
applications have been installed on the client machines, where they have been installed 
and the various components that are part of the installation. It contacts the application 
servers to get the AppInstallBlock. The AIM uses the AppInstallBlock to then make the 
appropriate calls to the file spoofer; to install some files on the local disk; to t4 warm" the 
cache and to update the start menu and other short cuts as needed. 

ECM 

The ECM is part of the ECE. Its goal is to: 

□ Handle all file requests from the EFSD, either by using previously cached 
contents or requesting the contents from a server. 
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□ Work with the LSM to insure that all applications have appropriately validated 
licenses before their files are accessed. 

The ECM handles the volatile and non-volatile eStream cache on the client machine. It 
performs demand fetching from the appropriate server(s). Based on the client's observed 
behavior, it compiles updated profiling data, which may periodically be uploaded to a 
server. 

EPF 

The ECM is part of the ECE. Its goal is to intelligently use prefetching of file data to 
reduce latency of pages requested from the EFSD; this prefetching may result from 
profiling data or heuristics. 

CNI 

The client network component is the common point of connection between the rest of the 
eStream client components and the various eStream servers. Any client module that calls 
an interface of a server does so through the network component. 

EMS 
CBM 
CIN 

The client installer is a simple InstallShield (or simpler) application that will install all of 
the required client software. 

FSP 

The purpose of the file spoofer is to redirect file system accesses from some non-eStream 
drive. This may be necessary in order to support applications running under eStream that 
are hard-wired to access files in a specific location. The file spoofer may also be used if 
we are interested in providing a version of some system file different from the one 
actually on the client machine. 

and ensure that these creates are redirected to a file we specify. The redirection could be 
to a file on the EFSD, or to another, non-eStream'ed file. 

EFSD 

The EFSD provides standard kernel file system interfaces to the I/O manager and other 
kernel-mode components. It works with the NT Cache Manager to efficiently cache file 
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and directory contents. Its view of the ECM is essentially like that of a disk driver, 
sending primarily read and write requests as needed. 

No Cluster 

The VM clustering disabling driver (aka NoCluster) disables virtual memory clustering 
under Windows. While we don't fully understand all the implications, using this driver 
substantially reduces the average file system paging request size and can dramatically 
improve performance of eStream, especially on slower connections. 

Virtual memory clustering, as implemented in Windows NT/2000, is intended to improve 
performance when paging to and from physical disks. If possible, we would like to 
disable clustering only for those threads/processes that will be doing a significant amount 
of I/O to the eStream file system. 

Server components 

The following are the server components for eStream 1 .0: 

□ App Server. This is essentially a file server for eStream sets. It satisfies requests 
for pages from eStream files from the client. 

□ Slim Server. This handles requests from a client for user and service provider 
information, and grants access tokens to the client for executing eStream 
applications. 

a Web Server. ??? 

□ Monitor. This enables an administrator to view the server components. It 
regularly pings the various servers, takes disabled ones offline, and adds new 
ones to the pools. 

□ eStream Database. This tracks all user information and server resources for a 
given service provider. 

App server 

The application server is there to handle read requests for files accessed by eStream 
clients. Any file accessed on a client through the EFS can have this read request passed 
to an app server. 

This will be the hardest working eStream server. It will respond to both synchronous 
clients, for many different types of applications and files within those applications. 

Slim server 

The Software License Management (Slim) server is responsible for: 
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□ Managing data related to users, the groups they belong to, and the applications 
they are subscribed to 

□ Validating the licenses for applications executing on clients 

□ Tracking all outstanding licenses currently in use 

ASP web server 

This describes only those interfaces on an ASP web server that relate to handling 
eStreamed applications. 

Logically, the ASP web server is the backend web interface for user requests— e.g., get 
billing information, subscribe to a new app, or request a list of all possible apps a user 
can subscribe to. In the current model, the web server doesn't actually handle these 
requests, but instead passes them on to the appropriate eStream-centric server. 

Monitor 

The monitor utility is responsible for monitoring the overall health of the system. It is 
responsible to report server status, server traffic, illegal access etc. It will ping the 
Application Server and the Slim servers to gather the statistics and display them. 

Database 

Builder components 

These are the builder components for eStream 1 .0: 
Q ??? 
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eStream 1.0 Requirements 




1.0 Introduction 

This document describes the high level requirements for the eStream 1.0 product. These 
requirements are given first as lists for the client and server components and then as 
scenarios. 

To facilitate the development of follow-on products, eStream 1 .0 does not include 
attributes that explicitly preclude future support of thin clients or of data ubiquity. 



2.0 Client Requirements 



The following are performance and functional requirements for the client portion of 
eStream 1.0: 



ID 


Description 


Priority 


1.0 


eStream client software operates on Windows 2000, 
Windows NT4, & Windows 98 for x86 clients. 


10 


1.1 


eStream client software may collect profile 
information during application execution; this 
information is used to improve client-based 
prefetching. This data is not uploaded. 


8 


1.2 


eStream client software supports eStream execution 
of top-selling (as represented by Ziff-Davis suites) 
desktop & laptop applications; these applications 
are listed in section 5.0 below. Other applications 
are supported (opportunistically) as well. 


10 


1.3 


eStream client software is obtained via the web or 
via some distribution media & is installed via some 
industry-standard [e.g., installshield] mechanism; its 
installation requires administrative privileges. 
Install reboot should be avoided, unless needed to 

eStream client software can be upgraded w/o 
reinstallation and w/o breaking installed apps. 


10 


1.4 


eStream client software operates across the different 
natural languages supported by Windows. The first 
release is an English-language release. 


8 


1.5 


Applications running under eStream have an ave 


8 
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interactive response time within 10% of client 
native for connections at 256K bps or higher. 




1.6 


eStream client software is able to operate with only 
16M of available disk space; this is the minimum 
supported configuration. User is encouraged to 
allow cache to grow beyond an arbitrary limit for 
best performance. 


8 


1.7 


eStream client software supports simultaneous 
execution of multiple eStreamed applications, 
including multiple instances of a single application 
(by a single user at a time; please see 1.21). 


10 


1.8 


eStream client software is able to unambiguously 
reference a particular ASP license for an 
application. 


10 


19 


Applications being eStreamed function in the same 
way that they would if they were installed locally. 


10 


1.10 


eStream client software tolerates server failure [i.e., 
it continues running any active apps and allowing 
apps to be launched], though possibly with some 
delay, assuming that an alternative server of the 
needed type is accessible. 


10 


1.11 


eStream client software detects and tolerates lost or 
garbled messages. 


10 


1.12 


It is difficult to steal an eStream application's code 
or data from the client. 


10 


1.13 


When an eStream application is being installed on a 
client, the process detects if the app is already 
installed & requests user confirmation to continue; a 
single version of an application is available to the 
user at a time. Install reboot should be avoided, 
unless needed to minimize potential 
stability/reliability problems. 


10 


1.14 


Upon uninstalling an application, application- 
specific changes to the client system are removed or 
undone. 


9 


1.15 


eStream client software makes minimal changes to 
the client system when running, avoiding/hiding 
any registry, DLL, & non-Z file system changes as 


10 


1.16 


eStream client software makes a run/no run license 
decision quickly enough when an eStreamed 
application is started not to cause customer 
satisfaction issues. 


10 


1.17 


eStream client software is launched/terminated at 
boot/shutdown. It is normally activated/deactivated 
at logon/logoff of an eStream user; it may 


10 
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temporarily be deactivated/activated at other times 
to allow specific administrative activities to occur. 




1.18 


User is able to set initial size of client cache. User 
is able to increase the ^i7p of thp cachp later without 
significant performance penalty. 


9 


1.19 


eStream client software does not include explicit 
ASP logon/logoff to run installed apps; ASP 

irlpntifipntirvn Hatn Qtnrpfl on pliant mnpHinp allnwQ 

llivlllllllsCHlvSli Uola alUlCU v/ll UJlCill UlaVlllllVs diJUVVo 

AccessToken to be obtained in seamless manner. 


9 


1 20 

1 -Z.V/ 


p.^trpJitn plipnt Qfyfrwnrp fViPilitatPQ rnamtno (\ p 

moving one's client system to a different site); 
eStream server info is not invalid/inappropriate 

wVlPtl tVlP pllPtlt 1Q TYlOVPfl tn 51 ffi "flfpfPTlt VPT111P 
W1IV11 lllt> IrJi&lil lo illV/VvU IU <x UlllVitlll VC11UC. 


in 

lv/ 


1.21 


Only one user at a time on a particular client can 
have eStream active. 


10 


1 00 


f>^tT£*dm will not o11/yv«/ nc^rc tr% nin tt-»*» comp 

coucdJii will nut oJ low UdCio tu run uic odJiic 
application from multiple clients simultaneously if 
the license prohibits it. 


in 

1 V/ 


1.23 


eStream client user interface will support HELP and 
aduu i iuncuons, mciuamg iiilks xo weDSiies wiin 
FAQs and support access information 


10 


1.24 


To start the process of having a particular ASP's 
application subscriptions known & kept updated on 
a client, one must visit the associated ASP website 
with that client at least once. 




3.0 


Server Requirements 




The following are performance and functional requirements for the server portion of 
eStream 1.0: 


ID 


Description 


Priority 


1.0 


eStream provides user and account management 
capabilities. 


10 


1.1 


User Account Creation/Deletion surmorted. 


10 1 


i.z 


Applications. 


iO 


1.3 


User is able to view billing and account 
information. 


10 


1.4 


User is able to change password/address/billing 
information online. 


10 


1.5 


User is able to list all available & subscribed 


10 
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applications. 




1.6 


User is able to access online help/doc, including an 
FAQ database. 


10 


1.7 


Omnishift provides interfaces to facilitate customer 
support by third parties. 

I-X i. hi 


10 


1.8 


User is able to enter/modify data securely. 


10 


1.9 


Both IE 4.0/later and Netscape Navigator 4.0/later 
browsers are supported. 


9 


1.10 


ASP agent [i.e., special administrative user at the 
ASP] is able to access all user information. 


10 


1.11 


ASP agent is able to disable a user. 


10 


1.12 


ASP agent is able to modify license information for 
a user. 


10 


1.13 


User is able to add additional users to the account. 


10 


1.14 


Web Server is highly scalable. 


10 


1.15 


Servers are able to operate in non-English language. 


9 


1.16 


ASP may operate eStream system with single 
server. 


9 


1.17 


Flexible access/export of billing information is 
supported, to facilitate 3 rd party billing systems. 


10 


1.18 


eStrpam <?prvpr <softwPirp anH pStrpam annQ pan hp 

VUUVlUJJ Ovl VW1 Jvll V V Cll \s CUiU vU U vOill CILILro \stXlX U t/ 

upgraded w/o impacting installed eStream client 
software All ur>t?rades are backwards comnatihlp 


10 


2.0 


The eStream framework [ASLM Server] provides a 
mechanism to validate the usage of application 
comoonents with resnect to billing models 


10 


2.1 


ASLM server is able to validate users to use 
specific applications. 


10 


2.2 


ASLM server records all usage activity down to the 
granularity necessary to support billing models. 
The granularity will be reasonably large. 


10 


2.3 


ASLM is able to release license on explicit remipst 
or timeout from the client. 


10 


2.4 


ASLM is portable across a wide variety of 
platforms and operating systems, including but not 
limited to: Windows NT4, Windows 2000, Solaris 
UltraSPARC, and Linux. 


10 


2.5 


ASLM servers are fault-tolerant. 


10 








2.7 


ASLM server is able to report Denial of Service 
attempts. 


10 


2.8 


ASLM server reports illegal accesses. 


10 


2.9 


ASLM is able to register its presence/load to the 
Web Server(s). 


9 


3.0 


eStream Framework provides management and 
monitoring tool (EMMT) to manage the servers. 


10 
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3.1 


EMMT is able to start/stop servers in the eStream 
framework. 


10 


3.2 


EMMT is able to monitor server activity for all 
servers in realtime. 


10 


3.3 


EMMT is able to configure the servers. 


10 


3.4 


EMMT is able to provide historical reporting. 


9 


3.5 


EMMT is able to display information graphically 
and in spreadsheet format. 


8 


3.6 


EMMT is able to raise alarms on predefined events. 


9 


4.0 


eStream framework nrovides a mechanism to 
deploy the application via the eStream Builder. 


10 


5.0 


eStream framework sunoort a varietv of licensing 
models. 


10 


5.1 


Floating license model is supported, 
n User -k Licenses 


10 


5.2 


Names User License model. (Special case n=k) 


10 


5.3 


Time based licenses at billing granularity. 


10 


5.4 


High water mark license. 


10 


5.5 


Node locked licenses. 


8 


6.0 


App Server is able to Authenticate client's accesses 

I vis* A pppcc i /"klfPTic 1 rr\mn1ptpl\7 1r\r* oll^/ 
^VJd /Al^l/Coo X (JlvClloy CL/IJI|JJClCljr lUCoJiy. 


10 


6.1 


App Server encrypts returned data (via a random 
key chosen by the client); it must be 
computationally infeasible to steal an application's 
code while it is being distributed or to determine 
which application a client is running. 


10 




/\pp oci vci lo do 2>idicieos ds possiDie io aiiow ciieni 
to switch to alternative app server w/o significant 
overhead. "Stateless" means that there is no server 

PfvntPYt tViat wnillH V>f» Inct if* tlif* Qf*r\JF>r wpnt rlrnx/n* 

wlll&Al Uldi WUUIU t/C JUM 11 L1JC oCl VC1 WCIll Ui/Wllj 

one classic example of this is that "file open" is 
recorded on the client, not on the server. 


i n 


6.3 


App Server is optimized to respond to requests with 
minimal server loarl thprphv maifimi7 , inof 
scalability. 


10 


6.4 


Ado Servers mav he prouned alonp with anv 

* X^/yj K-J vi » Vlij 1 I 111 J Ks\s glUUUUU CLl\Jllp^ Willi (UJ Y 

number of other such servers into a farm with 
minimal inter-server interactions (as to maximize 


o 


6.5 


Add Server communicates with clients thru 
firewalls. 


10 


6.6 


App Server communicates with clients efficiently 
(e.g., via persistent HTTP connections). 


10 


6.7 


App Server is able to install new eStream sets w/o 
having to go down. 


10 


6.8 


App Server is robust, able to run for long periods 


10 
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without crashes (i.e. no resource leaks, and handles 
most/all failure modes for system operations); 24/7 
operation. 




7.0 


App Servers, ASLM Servers, ASP Web Server and 
EMMT communicate through a database which will 
include but need not be limited to Microsoft 
SQLServer. 


10 


4.0 


Builder Requirements 




The following are performance and functional requirements for the builder portion of 
eStream 1.0: 


ID 


Description 


Priority 


L0 


The Builder installation monitor runs in the 
background, when an eStream application is 
installed as part of its preparation or building 
capabilities. 


10 


1.1 


The Builder installation monitor captures all the 
updates to the System Registry that take place 
during the install. 


10 


1.2 


The Builder installation monitor records all the files 
created in the two kinds of directories: the install 
directory and the common directories. 


10 


L3 


The Builder must be able to gather initial set of 
application profile data. This data consists at least 
of the page access pattern for starting and 
immediately shutting down an application 


10 


1.4 


The Builder must package the eStream Set into an 
easily manageable packages suitable for ASP 
administrators to download to their servers. 


10 


1.5 


The Builder must be able to collect per-user profile 
data from the Profile Server and merge the profile 
data into a combined data usable for updating the 
profile data in the appInstallBlock. 


8 


1.6 


The Builder should be run in an environment where 
no other applications are running. 

installation set(s) for each of the clients eStream 1 .0 
is going to support. 


10 

, ; 


1.8 


It should be possible to change the appld of the 
eStream set when an ASP wants to "install" the 
eStream set in order to host it. 


10 


1.9 


It should be possible to create a merged eStream set 


10 
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for a suite of applications. 




1.10 


It should be possible to test the eStream Set created 
by the Builder using a stand-alone tester and not 
require the eStream client+server programs. 


10 


1.11 


The appInstallBlock should have support for 
indicating upgrades at the support site 


10 


1.12 


In the process of creating an eStream set it should 
be possible for the user to delete file entries and 
registry entries manually to "trim" the eStream set if 
she so desires assuming the user knows what she is 
doing. 


10 


1.13 


The Builder should be run in a clean machine with 
as few software installed/upgraded as possible. 


10 


1.14 


The Builder should support individual applications 
in a suite even if the installer of the suite doesn't 
allow installation of individual applications. 


10 


1.15 


The Builder must be able to create an initial set of 
cache contents for the eStream client and allow the 
initial size to be selectable by the user or 
automatically. 


10 



5.0 Client Use Cases 

5.1 USE CASE: Installation of eStream client code 

• Obtain eStream client code bits* 

• Install z: file system hooks & setup to have z: mounted at appropriate time. 

• Install eStream client code, which services z: file sys requests from local cache or 
from servers & which handles sideband communication w/ servers, and setup to 
activate estream client code at time desired by user (boot, login, on demand). 

• Install NoCluster.sys to disable page fault clustering at system boot. 

5.2 USE CASE: Installation of application 

• Obtain AppiU dc App Server name lor installation from SLM Server. 

• Download AppInstallBlock information. 

• Perform initial installation & setup for app, after checking system for previously 
installed version of app & issuing any appropriate warnings. 
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5.3 USE CASE: Uninstallation of application 

• Remove all registry/DLL/filesys changes associated with app installation. 

• Remove all other data associated with application. 

5-4 USE CASE: Uninstallation of eStream client code 

• Remove z: file system hooks, eStream client code, & nocluster.sys. 

5.5 USE CASE: Execution of eStream client code 

• Respond to z: file sys requests and detect when new eStream app is referenced. 

• Support Client UI requests. 

5.6 USE CASE: Execution of application 

• Obtain Access Token & list of App Servers from SLM Server. 

• Contact App Server(s) as desired to obtain file system data. 

• Respond to running application's requests, collect usage data. Cache portions of 
application, file system info, & user preference info. 

• Detect server connection issues (apparent loss of connection or connection response 
below acceptable threshold) & licensing issues; negotiate with ASLM Server as 
needed. 



6.0 Server Use Cases 



6.1 USE CASE: Create an Account 

• Customer brings up browser and connects to ASP Web Server 

• Screen disnlav shows "create account", customer selects and enters reouired account 

iijUO AnxKS y Uov.j.iu, pVVWiU, ClUj 

ASP Web server writes account info to Acct DB using Add Account where a unique 
Account ID is assigned 

• Account ID is returned via web page 
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6.2 USE CASE : Create a User 



• Customer brings up browser and connects to ASP Web Server 

• Customer enters their userid and pword 

• ASP Web server contacts Acct DB, using AddUser userid and initial password, gets 
Acct info and displays to Customer 

• Customer selects "add user" and enters required user info (username, address, email 
etc) 

• ASP Web server writes user info to Acct DB updating account info 

6.3 USE CASE: Modify Account 

(includes disabling an account or user, removing users from accounts, changing pwords 
etc) 

• Customer brings up browser and connects to ASP Web Server 

• Customer enters their userid and pword 

• ASP Web server contacts Acct DB, passes along userid and pword, gets Acct info 
and displays to Customer 

• Customer selects "update info" and enters desired changes 

• ASP Web server writes updated account info to Acct DB 

6.4 USE CASE: AddSubscription 

• Connect to ASP web server 

• Enter account number, username, password 

• Verify that user is account admin using GetUserPermissions 

• Get list of possible subscriptions (using ListPossibleSubscriptions) 

• Get list of current subscriptions for account (using ListCurrentSubscriptions) 

• Display in page - User chooses a subscription and license type 

• Display a screen to allow the user to configure the license. For a floating license, 
allow selection of users, etc. 

• Call CreateSubscription to compose the new subscription for each user and create 
licenses. 



6.5 USE CASE: Building an eStream set: 

• Start w/app CD-ROM, and a freshly installed OS (plus latest service pack?). 

• Install app into Z: drive (could just be a regular network drive) 
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• A special system monitor logs all registry changes and file system changes 
during the install. 

• File system changes to C: during install probably need to be spoofed (or have a 
registry entry point to Z: instead), especially newly added directories, so need to 
do the appropriate thing. 

• From this log and the actual files as installed on the machine, the eStream set 
builder creates the eStream set, which is a small set of related files. 

• Separately, we need to actually set up the app for eStreaming, then run it and 
collect profile data to seed the initial page prediction map. 

The AppServer UI (interface to user to control an Application Server on a particular 
machine) presents the following management functions: 

A. Starting a server: 

• AppServer UI always indicates whether an AppServer process is up and running 
(and alive w/status), and if present prompts for restarting the current server 
process. 

• Otherwise it goes ahead and starts Uj>the AppServer process and reports any 
errors. 

B. Stopping a server: 

• Simple, just stops any running servers, gracefully, perhaps prompting user for 
ungraceful shutdown if not successful. 

C. Install eStream set: 

• Each server is configured with a specific eStream set directory, under which it 
places (in their own individual directories) the actual eStream set contents (a few 
files on the native file system). 

• User indicates to AppServer UI where to find the eStream set package provided 
by Omnishift. AppServer UI authenticates the package, and verifies its integrity, 
and if successful, unpacks and places the constituent files in the server's eStream 
set directory. 

• Note that it is possible for the eStream set directory to live on a file server shared 
by other Application Server machines, so installation may be required only once 



is responsible for replicating eStream sets across a farm to ensure the farm 
machines are symmetric). 
• How does the server know a new eStream set is available? Each set is assigned a 
VolumeBD, and the set contents can be placed under a directory with the same 
name as the VolumelD. The install is synchronized via a AppList file, which just 
lists the valid VolumeEDs, which the Application Server only reads, so an entry 
is added at the end of the install procedure. The AppServer UI then must send 
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some kind of message/signal to the Application Server to have it resync with the 
file (and start serving the new app). 

• Also note that eStream set install is doable without bringing down the server (or 
any server in the farm. 

• Having done this, it will probably be necessary to notify the SLM and Account 
Servers that a new app is available. With some scripts provided by omnishift, this 
sould be done by a human administrator. They need to know the VolumelD of the 
app that was installed along with the foil name* so that the client can initiate an 
app install procedure via the VolumelD (the server can then provide the 
AppInstallBlock which probably has a fixed reserved global FilelD). 

• Questions: What if app is already installed (want to allow reinstall or force 
remove first)? What if app is being upgraded (probably also should be a remove 
and then install)? 



D. Remove eStream set: 

• First we probably have to disable the application on the SLM/Account servers. 

• This probably will require sending some kind of message to the Application 
Server (if running) to stop serving the given eStream set, and then waiting for any 
active connections to expire. 

• Then we can just remove the entry in the AppList file, and delete the file system 
image. 

E. Configure Application Server: 

• The AppServer UI presents various configuration options to the user (stuff like 
logging, port #, threads, etc.) Some may require restarting the Application 
Server to take effect, others may take effect immediately. 

Another activity that occurs, automatically, is the processing of profile data. It is not 
clear what the page prediction map looks like, but clients will periodically send profile 
data to the Application Server, which aggregates it, and must store it persistently, and to 
allow new clients to benefit from improved prediction. There may need to be a special 
module that can take the aggregated profile data to modify the prediction map. 



Where are we? 
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• Client PC has installed the subscribed app and has received a subscription token, 
and the name/IP of SLM Server. 

• Customer is accessing an app file and doesn't have an access token for it, yet. (i.e. 
double clicking z:\word.exe). 



Players involved: client - cache mgr, SLM Server and indirectly, 
User/Account/Sub/Rights DB. 

What happens: 

• Client contacts the SLM server and gives: subscription token, user/passwd. 

• SLM server looks into the user/acc/sub/right DB to 

o Authenticates user and password; may return: "invalid user". 

o Authenticates subscription token; may return: "invalid subscription token" 

o Look at the Accounts container and see if any licenses are available. If so, 

check it out by creating a new access token and updating the accounts 

container. It may return: "can't get license". 

• Return an access token to the client and a list of app servers. 

6/7 USE CASE: Process File Request - steady state 

Where are we? 

■ Client has installed the app and has a list of app servers, 

■ Client is holding a valid access token that it acquired from the SLM server. 

» Client, while processing an IRP, needs to access portion of file on the app server. 

Players involved: client - cache mgr, app server. NO SLM server or no 
user/account/dub/rights DB. 

What happens: 

» Client contacts one of the app servers and gives: access token, App ID, File ID, 
length and file offset. 

■ Add server quickly verifies the eviration rl^te on the access token. 

<u H iiAuat uui iiCoU to Contuse iiic U^Ci/aCCOUril/SUL)/ HgiltS Ui> CO GO UilS. 11 

only cares about the time-validity of the token. If token has expired, return 
some kind of an error back to the client. 

■ App server locates the data and sends it back to the client. 

NOTE: we are simplifying this quite a bit when discussing the 
scenarios because we are not sure exactly how we are going to 



Omnishift Confidential 



Page 12 



manage the server farms. Another key question is MUST all 
app servers host all estream apps ? 



6.8 USE CASE: Renew an Access Token - steady state 

Where are we? 

» Client acquired an access token from the SLM server. 

■ While running the app, client sees the needs to renew the access token. This may 
happen synchronously when the user touches one of the app files, or by a timer- 
driven client daemon that periodically renews an access tokens before it expires. 

Players involved: client - license manager, SLM manager, and indirectly, 
user/accounts/sub/right/ DB. 

What happens: 

■ Client sends an access token to the SLM server. 

■ Check the time-validity of the access token. 

Assumption: SLM server assumes that only valid access tokens can be renewed. 
An expired token implies a lack of renewal, which implies releasing the license. 
SLM server can try to acquire the license, but there is no guarantee that it will 
succeed. 

o If token is expired app, goto Scenario: Acquire Access Token. 

■ SLM server accesses the user/account/sub/rights DB to: 

o Generate a new token that will expire some time in the fiiture 

(configurable parameter), 
o Update the account container in user/account/sub/rights DB. 

Return the new access token. 

6.9 USE CASE: Validate user request for access to an 
application server 

Procedure: 

Query AccountDB for license to access application appED in subscription subID 
If (no valid license) then 

Send FailureReason to Client 
Else 

Send accessToken, appServers to Client 
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6 AO USE CASE: Add subscribable application from an 
account 

Interface Required: 

SLMServer: :AddSubscribedApp(accountID, subID) 
Procedure: 

Receive accountID, and subID from the Client 
Check for valid accountID, and subID on AccountDB 
If (no valid accountID or subID) then 

Send FailureReason to Client 
Else if (subID is not already subscribed under accountID) 

Add Subscription subID to Account accountID in AccountDB 
Send Success to Client 



6.11 USE CASE: Remove subscribable application from an 
account 

Interface Required: 

SLMServer: :RemoveSubscribedApp(accountID, subID) 
Procedure: 

Receive accountID, and subID from the Client 
Check for valid accountID, and subID on AccountDB 
If (exist subID in accountID) then 

Remove Subscription subID from Account accountID in AccountDB 

Send Success to Client 
Endif 

Send FailureReason to Client 



6.12 USE CASE: Monitor/management tools 

Interface Required: 
SLMServer: :GetTrafficHistory() 

SLMServer: :GetUsageInfo(userID, appID, subID, accountID) 
SLMServer: :GetCurrentTrafficO 
SLMServer: : AddServer(serverID) 

SLMServer: :RemoveClient(userID, serverlD) 
SLMServer: :GetErrors() 
SLMServer: :DumpErrors(filename) 
SLMServer: :DeleteErrors() 
AppServer::GetTrafficHistory() 
AppServer: : GetCurrentTraffic() 
AppServer : : GetErrorsQ 
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Procedure: 

SLM Servers keep track of traffic info. The monitor/management tool can query 
the SLM/App Servers anywhere for traffic info. Some examples of traffic data: 

- Traffic history of particular server on number of clients served per unit time 

- Monitor length time a userlD used application appDD under subscription subID and 
charged to accountID 

- Monitor current load information on all servers (SLM server and app server) 

- Allow admin manually add/remove some servers from the pool. 

- Allow admin to kick some clients off the server. 

The monitor/management tool can also be used to display a list of errors logged by the 
servers. 

- Monitor errors and be able to categorize by error type 

- Monitor errors occurring between certain time periods 

- Monitor errors reported by a particular server 

- Manage errors to dump the errors to a file 

- Manage errors and delete a subset of errors 

Finally, the monitor/management tool can check for any illegal accesses. 

- Monitor failed attempts to access SLM Server with bad password, especially on 
repeated failed attempts in a short time frame. 

- Monitor any attempts to use a particular license and failed. 

- Monitor access to SLM Server from non-typical IP addresses for a particular 
account. The server is required to save the history of IP addresses of accesses to 
a particular subscription account. 



6.13 USE CASE: Adding a new application server. 



Summary: 

An application server's functionality is to provide applications eStream sets to client 
application. An application server is generally added to the system to provide greater 
scalability and/or to provide additional application support. 

Actors: 

2. ASLM Server(s): The ASLM server needs to be notified of the presence of an 
additional application server and the services it provides. 

Inputs: 

1 . Application server(AS) installer 

2. Application eStream sets. These may be available from one of the following 
location: AS installer, some other AS or Farm Manager Server(some central 
repository) . 
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3. SLM Server location(This input may not be required based on scalability solution 
that we decide on). 
Processing: 

1 . Using the AS installer install the application server. 

a. <Server install use case to be added here? Later> 

2. Copy the Application eStream sets. There are several options here: 

a. Provide the eStream sets as a part of the installer. 

b. Provide a script to ftp to another Application server and copy the eStream 
sets. 

c. Provide a management tool to manage the copying of the eStream sets. 
From the ASP's perspective this is the best solution. A tool which 
provides tracks the application would be useful to manage the load. 

3. Configure the server. The server needs to know the additional application sets that 
it supports(? This may not be required). 

4. Start the server. 

5. Register the server with other SLM servers. The following options apply: 

a. Multi-cast the "new server and services" message to the SLM servers. 

b. Register the server to a local object server which in turn notifies the object 
servers across the system. CORBA model supports this. 

c. Using the resonate model(described below), all appservers are essentially 
the same server, ie Address app.foo.com will point to a set of app servers. 
A new server enabled will resonate software will automatically register 
itself with the resonate scheduler. (How do we make the resonate 
scheduler aware of the applications available on the app servers?) 

Outputs: 

1 . The App server is installed and running with a set of applications available on it. 



6.14 USE CASE: Removing an Application Server. 

Summary: 

An ASP administrator may decide to remove an application server from the system for 
various reasons. Removal of server from the system would result in notification to the 
rest of the SLM servers that it is no longer available for servicing the objects. 

Actors: 

2. SLM Servers. 
Inputs: 

1 . Application server running on the machine. 

2. ASLM Server(s): The ASLM server needs to be notified of the presence of an 
additional application server and the services it provides. 

Processing: 



Omnishift Confidential 



Page 16 



1 . Stop the application server. This will result in the Application server informing 
the rest of the ASLM servers that it will no longer take any requests. This in turn 
may result in an application being unavailable for usage. Depending in the 
framework used, this can be done in one of the following ways: 

a. Multi-cast the message to the ASLM servers. 

b. Just stop the server in the CORBA framework. The local ORB server will 
notify the unavailability of the resource to the rest of the framework. 

c. Using the resonate model to scale would imply that you just stop the 
server the resonate agent on the server will notify the resonate scheduler to 
deregister the servers. (However its not clear if you can also deregister the 
objects served by the server.). 

Outputs: 

I . ASLM servers are notified of the removal of the resource. 



6.15 USE CASE: Add a new ASLM server. 



Summary: 

The ASP provider may decide to add an additional ASLM server to enhance the 
performance of the system. The additional ASLM server added to the system should be 1 
accessible to the ASP's Web Server so that it can direct the clients to the SLM server. 
(This may not be required if we deploy the Resonate model of scaling). 
Actors: 

1 . The ASP administrator. 

2. The ASP Web server. 

Inputs: 

1. ASLM installer 

2. Web Server location(This input may not be required based on scalability solution 
that we decide on). 

Processing: 

1 . Using the ASLM installer install the application server. 
<Server install use case to be added here? Later> 

2. Start the server. 

3. Register the server with ASP Web Servers. The following options apply: 

a. Multi-cast the "new server and services" message to the Web servers. 

b. Register the server to a local object server which in turn notifies the object 

c. Using tne resonate modeKdescnbed below), all ASLM are essentially the 
same server, ie Address aslm.foo.com will point to a set of ASLM servers. A 
new server enabled will resonate software will automatically register itself 
with the resonate scheduler. 

Outputs: 

The ASLM server up and running. 
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7.0 Builder Use Cases 



7.1 USE CASE: Install Monitoring 

• Query builder for CD media and installation executable(s) 

• Monitor various registry and file updated during installation 

• Merge installation data for all applications in a suite 

• Relocate files from C: to Z: directory 

• Create appInstallBlock and package the appInstallBlock with the application files 

7.2 USE CASE: Profiling 

• Query builder for application executable(s) 

• Monitor sequences of file accesses from OS to the file system as profile data 

• Identify the subset of the profile data as the initial cache contents 

• Merge profile data and initial cache contents into the corresponding 
appInstallBlock 



8.0 Key Applications for eStream 1.0 

Winstone99: 

Business: QuattroPro, WordPerfect, Lotus® 1-2-3, Word Pro, 

Access, Excel, PowerPoint, Word 
High-end: Adobe® Photoshop, Adobe® Premiere, Microsoft® FrontPage, 

Sonic Foundry® Sound Forge 
Content Creation Winstone 2000: Macromedia Director, MacromediaDreamweaver 

Please note that release of Business Winstone 2000, which was originally slated for 
6/27/2000, has now been postponed until the Fall Comdex & will be called Winstone 
2001. As soon as the contents of this suite are released, we should move quickly to 
assess our support for its application set. 
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eStream 1.0 Server Scaling Estimate 



Anne Holler *\ 




! Version 1.0 



Introduction 



This document presents an estimate of server scaling for the eStream 1.0 product as 
compared with its chief competitor, the Citrix product as deployed by Personable. The 
document presents relevant attributes of the basic application execution model for each of 
the two products, discusses and gauges the impact of the areas in which server scaling 
differs between them, considers the effects of additional attributes of the two products on 
server scaling, & finally summarizes the differences in expected server scaling in terms 
of a number. Please feel free to challenge the assumptions, methodology, & calculations 
herein, now & as we move forward through the design & implementation phases. 

In the process of developing this server scaling estimate, certain assumptions about user, 
system, & program behavior are made, & certain design/implementation goals of the 
eStream 1 .0 product are assumed. This material is listed in separate sections at the end of 
the document for ease of reference. 

This work does not intend to imply that the user experience of the eStream 1 .0 & 

Personable/Citrix products is expected to be comparable with respect to the relative 

server scaling point identified. Interactive response differs between the two products; 

first-hand experience with server-based applications running on New Moon & 

Personable/Citrix and client-based applications running on the eStream prototype 

suggests that the former are sluggish on an ongoing basis with respect to activities such 

as selecting from pull-down menus & the latter are as responsive as native wrt such 1 v • • ^ 

activities, with noticeable delays engendered only when heretofore unused portions of the 

application's functionality are exercised. Reliability also differs between the two 

products; smooth fail-over of an active Personable/Citrix application to another server is 

not supported, whereas such fail-over is included in the eStream 1.0 design. 

This document does not address an area in addition to server scaling that may be of 
competitive interest to ASPs; that area is network bandwidth differences between 
eStream 1 .0 & Personable/Citrix. Though it might seem intuitive that sending application 
pages across a network consumes more bandwidth than sending user input & display 
output, aggressive client caching ameliorates the traffic associated with application 
oa<?7rts. whereas the traffic associated with tb*» H^r>1av rmtni.it can He ouite substantial 
accoruaig iu n<u woou s oou* Win i i ciiiiUiuj oci ver 6c cimx ivietairame ' [Hereafter, 
HARW99]. It may be worthwhile to collect and compare bandwidth data wrt the two 
products. 
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Application Execution Models 

PersonabWCte Client/Server Applicalion Execution Model 
eStream 1 .0 Client/Server Application Execution Model 

Application Server Scaling Comparison 

^impact onservers^^ 

server, is expected to be large. ^ interrupts such as mouse ft 

executing applications, *ff^%*%^ power for running applications ,s 
keyboard events. According to ' HAKW W. pro SF reinforced by the variance in 
typically the biggest Citnx P-f^^^t ^ wrt another ASP [An*], for 
Citrix scaling numbers reported wrt ^^xecution overhead. HARW99 
which the main differences se « * * ■» ££££ proce ssor, largely due to execution 
indicates that Citrix scales * 10 to «*g™^ P fauUs & accessing application 
overhead (though some overhead is due Ac M^cMB PS t tQ ^tand 

data, which is considered m the nex ^ sense m m infinite number 

how to model the relative benefit fo ^ actually using to execute'.); it 
of applications can run on a P rcc ^^^ from this factor as we do 

seems conservative to assume we get at ug have ^ factor do uble 

mimpactonservexscalmgofpr^i^ 

reads on the client in most cases, rather than on £ server,, ^ ^ ^ 

r ^ — — - u^'tTfrom disk whether the request came from server 

that the server overhead to fetch a page ^^J* we ^ constr uct file server 
execution or from client request, is compamb^ MAM gn^ ^ ^ ^ ^ 
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overhead for application file accesses is estimate to be £. rate . 

reduction in the number Ration server can handle 50 

Assuming a client cache miss rate of 2 A J* r overhead . Doubling that 

times as many clients with respect to this attribute of serv ^ ^ ^ 

amount due to the factor described m the W ^ as each 
eStream application server can handle 100 times as many 
Personable/Citrix application server. 

component of server scaling between dWBU and r ^ ^ ^ 

consents associated with executing ™ STappifcation server. 
^/^'f^^^S^^ted to support client 
Depending on the kind of application^ sign m [discussed in the 

displays, but HARW99 identifies netivo* and™ »hover dered by & 

Introduction section] - not «™ se ^er network overhead, but dre 

traffic. Each eStream 1.0 client gener M „ more clienK t0 be 
reduction in server load due to client apphcation ex«artlon ^ 
connected to a given server, network interface 

remove this overhead. 

Additional Server Scaling Considerations 

For eStream, W*^"**- " 

the client the contents of the ^^^^^Z information, etc. 
information, initial cache & profile A ^^ T f\ s ^ Uion is expected to be an 
Personable/Citrix does not have ^P^^ggest/provide mechanisms to 
infrequent occurrence and <*^"Jg?^ particular points in time, including 
smooth out (or specially hand e) ^P£<£^£fhi. (Managed) overhead 
product or application l"^ 1 ^^ cac he miss. Adjusting the 

S^yclSsas 
dc>esn 0 tha^ 

Given that eStream 1.0 i. ^^g^t?^SSUly. Let us assume 
wrt a comparably warmed chen « ^ 0 5% cache miss rate; the intuition 

that prefetching adds the equivalent mISS es occur (i.e., when we 
here is that prefetching is ^^^S^Sbefiw) and that we get unneeded 
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means tha, an eStream application see, can handle 50 toes as many M as a 
Personable/Citrix application server can. 

the application server. Both have an ASP Web Server pan 

services, from which a user can ^^f^SjS^^i^ 

applications, subscribe to nw fltoW-, ^ y D comparable complexity. Both 

in some way with an account database of pre^naW ^comp ^ ^ tQ ^ 

eStream & Personable/Citnx ^™£™Z^J$U in eStream 1.0, this 
an application, which is expected to ^e^ 1 ^ 0 However , 
process involves getting an AccessToken fro , m ^ ^ renewing m AccessToken. It 
Personable/Citrix does not have eStream f f'l"^^^^ does not add 
is expected that me eS^ 

significant overhead to the eStrean • 1 ADRM serv* U» . g 

granularity & AccessToken renewal frequency by en & , for token 

pWbyhavmgsom^ 

cancellation in the event of client *™r^^ip^ to a server; it is not 
records containing application profile "^mation being p ^ has 

known that Personable/Citrix has Fe ^ fences, etc). It is 

much other user information recorded at foe server impact for handling this 
expected that eStream ^ ^f^^f^Z^i?^^.} 

t^rjy^^^ 

eStream 1.0 & Personable/Citrix. 

Conclusion 

Bas «, on the disenssions in ^Pjevions section, ££*ES5E£ 
significantly higher ^^^f^^JZ^^^M 
cltent caching benefits, appl.cat.on ;^^ x ^! d 6 7 toes higher Sran the 
eStream server scaling » peraonablJcto application 

Assumptions Underlying Server Scaling Estimate 

Usersapplicationnsagep^^^^ 

change materially depending on whether s/ne.s ^ . „ ^..^ .o.u.u.a.eu 

- tote^ SSTSW might be interesting to see if oar 

=:SriT '. 0 may continue ,0 a„ow clients to access the 
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additional eStream ctan. Howev«, « s poss A * ^ wth (he 

collapse could occur on eSlreanr, >f cue* to*> we p respons e caused an 
"pa* of eStteam-s **^^£XL clients retrials. This sttuatron 

esrrean, !.0 clienU are fa, enough .0 ahow 

cScache miss goal of 2%. ^^"S ^caches would increase server 
document. 

Design Goals Snpporting Server Scaling Estimate 

Smrance goals also reinforces (he need fa a low eta ^ ^ 

E data indicating how hnge usa| , e B represented by the 

^cSSdr^ecriveoachonaanagenren.Pohc.ea. 
performance goals. ^ 
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eStream File System Straw Man Proposal 

VersionjlS 



Purpose 

The purpose of this document is to present a concrete proposal for the functioning of the 
eStream file system. In many places, I make some sweeping generalizations about how 
things should work without describing the data structures and interfaces involved in 
implementing them. This document should eventually involve into a design 
specification. 

Issues Not Covered 

This document does not attempt to cover all issues present in designing the eStream 1 .0 
product. In particular, the overall authentication/licensing/security architecture is not 
covered in detail here. It is expected that the security functionality will be mostly 
orthogonal to the design of the basic file system functionality. 

Background 

There are a number of different networked file systems out there. Many of them share 
some requirements with eStream. For example, AFS performs client-side on-disk 
caching, while Coda handles serious server redundancy and disconnected operation. 
Personally, I believe that AFS and Coda are the file systems whose designs are most 
relevant to us. For those interested in further background reading, you might also want to 
look at papers covering NFS, CIFS, xFS, DFS, and Zebra. 

Single File System Name Space 

Many modem distributed file systems present the network file system as a single tree 
mounted at some location on the client system, regardless of which server hosts the data. 
(In fact, with AFS, every file on every server in the world can be accessed through a path 
starting with /afs on the client, assuming the client can reach that server and has sufficient 
privileges to do so.) Compared with systems like NFS and Windows sharing, where each 
share is mounted in a different location on the client, the single name space provides 
greater ease of use. 

The eStream file system would present one universal logical file system. Regardless of 
which ASP provider supplies a particular volume, that volume will always be referenced 
via the same path on the eStream file system. That this is desirable or even feasible is 
predicated on the assumption that OTI is the only entity providing all eStream sets. Each 
volume must eet a unique identifier and a unique location to be mounted in the file 

those volumes must be identical. This way, we don't have to tag things in the cache 
based on what ASP they came from, and the cache manager doesn't need to know 
anything about ASPs. If done correctly, only the client networking component and the 
LSM need to know about ASPs. 

Volumes 
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A volume is a complete subtree of a file system. Volumes may contain files and 
directories. Volumes may not be mounted in other volumes. A volume is a logical 
grouping of files within the file system and is the unit of replication across servers. An 
application will reside in a single volume. Two applications will never share a volume. 

Volumes are uniquely identified by a 32-bit volume identifier. Each volume additionally 
has an 8-bit version number. This version number is incremented each time any file 
within the volume changes. (See supporting upgrades, below). Note that the volume id 
is globally unique. If two ASPs provide volumes with the same volume number (and 
version), they have identical contents. 

A volume may be replicated on any number of servers. Each SLM server contains a map 
describing the application servers that currently provide each volume. This global 
replication of this table is acceptable because volumes are added or moved infrequently. 

Identifying Files 

Files and directories are uniquely identified by the pair (volume id, file number). This 
tuple is called a file id. Volume id and file number are each 32-bit signed integers. 
Negative values for both volume id and file number are reserved for special purposes, 
leaving us with 2 A 3 1 possible volume IDs and 2 A 31 possible files per volume. 

Finding an Application Server for a Volume 

The SLM will tell the client which application servers currently provide each volume. It 
maybe necessary for the client to periodically poll the SLM to get up-to-date information 
about the state of the application servers. The License and Subscription Manager on the 
client will keep track of the currently subscribed applications and the application servers 
for each of these applications. 

Directories 

Directories are specially formatted files that are used in a special way by the file system. 
They are identified by file ids, just like other files. From a client-server point of view, 
they are read by the client in the same way as other files. Directories contain arrays of 
entries with the following format: 
( volume number, file number, flags, length, filename ) 

The volume number and file number are 32-bit signed integers. The flags are 32-bits of 
flags. The length is 16 bits and is the length of the filename in bytes. The filename is a 
non-NUL terminated Unicode string. The structure is padded with enough Unicode NUL 
characters to make the structure a multiple of 32 bits long. The next directorv entry 



The access token is not part of the directory, as a single access token is required to access 
all files in a particular volume. 
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The volume number is required so that the the client can construct a local directory for 
the root of the directory structure in the same format as other directories (see filename 
parsing below). It also helps to provide a sanity check. 

Accessing Files 

Assuming that a client has a file-id for a file that it wishes to access, the following client- 
server actions must be supported: 

For stat-like information on the file, we need a GetFileMetadata() interface. The client 
would provide the file id it is interested in and the proper access token for this file. The 
server will either return the metadata for the file or an error condition (like access token 
expired or incorrect access token.) The metadata contains the standard Windows 
metadata information, including file length and file access times. 

On a file open (CreateFile in Windows terminology), we need to verify that we have 
access to the requested file. This is probably best accomplished by calling 
GetFileMetadata and verifying that we can get the metadata. This way, we can fail file 
opens gracefully if we don't have an access token. 

On reads (and writes, when we support them), the client will send the file id and the 
access token to the server along with an offset and a length for the read and write. The 
server will respond with the data. Note that the same mechanism will be used for reading 
both files and directories. 

Pseudodirectories 

For those parts of the eStream file system name space that do not belong to any volume 
(such as the root of the file system), the client must construct appropriate directories 
based on the currently installed applications. This is to support filename parsing starting 
at the root of the directory. For example, if the client has word installed with a root of 
. /W orddir and it is volume number 3 and Photoshop installed with a root of /Photodir and 
is volume number 4, the client would construct a directory for the root of the entire file 
system containing 

File name, Volume number, file number 
"Worddir", 3, 0 
"Photodir", 4, 0 

(The file numbers are both zero here because 0 is the index of the root directory of each 
volume, and these are the mount points for each volume.) 

to reflect the newly installed apps. 
Filename Parsing 

Filename parsing is handled one element at a time, starting at the root of the file system. 
Parsing one path name element involves reading the parent directory's contents (from the 
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cache or the app server), searching it for the file matching the next path element's name, 
and getting the appropriate file id so it can do further lookup. 

Volume Versioning... Without File Versioning 

We can provide volume versioning and incremental volume updates without versioning 
each file in the file system. When a new volume is to be provided, we can append any 
new or changed files as new files in the volume, with new volume IDs that weren't 
already present. If a directory's contents have changed, then a new version of this 
directory will be built, with a new file number. This process will proceed from the leaves 
all the way to the root of the file system, eventually resulting in a new root. The old 
versions of things would still be available for old clients to access, but clients wishing to 
access the new version will simply start at the new root, and would thereby get to a 
consistent picture of the volume. Any file or directory that has not changed from the old 
version to the new one need not be replicated, and will be referenced by its old file 
number. (I.e. newly reconstructed directories will contain the old file number for any 
files that haven't changed.) 

If we reserve the first 256 file ids for the root directory, then the version number can be 
the same as the file number for the root directory. 

Note that if we decide that the complexity of this approach is too high, this does not 
preclude always creating a new volume from scratch for each update. 

Constructing File IDs 

It is the job of the builder to produce the volume file to file id mapping and to construct 
all of the directories. Because directories are files identified by file id, this process must 
begin at the leaves of the volume and proceed to the root. 

Note that constructing a new changed volume will consist of finding the diffs between the 
two volumes and producing some new directories. Changed or newly added files will get 
new file numbers, leaving the old ones around. Note that any directory that has had any 
descendents changed must be reconstructed with the new file numbers, and the new 
directory will get a new file number. This process will proceed to the root of the volume, 
which will receive a new file number. 

Server Failover 

All app servers for a particular volume must share the same mapping of file ids to file, so 
server failover is trivial. There might be a performance impact if the new app server 
doesn't have the requrested file in memory. 

Writing Files into the Application Install Directories 

Two approaches have been discussed for the problem of applications that want to write 
files to their install directories. First, this can be handled wholly inside of the eStream 
file system. The cache manager could allow writes to files handled by the eFS, but these 
writes would not be written back to the server. Instead, they would simply be written to 
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the eFS cache and marked non-purgeable. This approach's primary advantage is that it 
does not rely on a file spoofer. 

The other approach is to use the file spoofer to spoof some accesses to the z: drive. Any 
open for read/write access would cause the existing file (if any) to be copied to a location 
on the c: drive, and the file spoofer would then redirect the open to the newly created file. 
The file spoofer would have to keep track of any file created via this copy-on-write 
mechanism and redirect all future accesses to the copy. There are some issues to this 
approach. For example, it is extremely wasteful when files on the z: drive are opened for 
read/write access but are never actually written. However, it does help reduce the 
complexity of the eFS cache, and is trivial to implement if we have to do c: to z: file 
spoofing anyway. 

In either case, to support the creation of new files in an application's install directory, it 
must be possible to modify the contents of directories in the cache. 

If we don't use the file spoofing approach, there is the issue of how we support written 
files when we move to a newer version of a volume. It would probably be necessary to 
walk the cache and make sure that each written file gets placed in the appropriate place in 
the new volume version. This is likely to be non-trivial, because we need to have full 
information about the location of each modified file in the file system tree, and would 
need to download enough of the new volume directory structure to place these modified 
files there. 

64-Bit File Access? 

One question we should answer is whether we will support file sizes greater than 2 GB on 
the eStream file system. I'm inclined to say that such support isn't a requirement for the 
1 .0 product, but I also think that the implementation and verification complexity of 64-bit 
file access on the file system is low enough that we might want to consider building it in 
anyway. 

Simplifications 

We could preclude the possibility of an application consisting of more than one volume. 
Future Possibilities 

Epicon seems to make a big selling point of their technology involving "self-healing" of 
damaged application files. Such support could be provided by computing checksums on 
files in the cache. Whether or not we want to support this is an open question. My 
feeling is that it's something we should leave out of 1.0. 

Outstanding Issues 

Cache organization has not been addressed. 

Finding and downloading the app install block has not been addressed. 
Security in a multiuser system has not been addressed. 
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eStream Requirements from Multiple Perspectives 



Ricky Benitez 




1.0 OTI Requirements 

The following requirements are primarily driven by OTI and its business and operational 
needs. 

1.1 Functionality 

• The system can deliver applications to x86-based Windows NT 4, Windows 2000, 
Windows 95, Windows 98 and Windows Me systems clients in order to capture a 
substantial size of the existing client market 

• The client and server software, software upgrades and application sets can be 
delivered to customers electronically via the internet (ftp or web) to reduce the 
costs associated with the delivery of the product 

• The client and server software, software upgrades and application sets can be 
delivered to customers on traditional media (CDs, DVDs) to increase the speed of 
service at remote sites and to provide the flexibility of servicing isolated networks 

• The system allows Omnishift to perform audits, with the permission of the ASP, 
to ensure that we are being appropriately paid per our license agreement with the 
ASP 

1 .2 Localization 

• The language used by the server components should be specific to an 
administrator to facilitate the installation and remote operation of a foreign-run 
server by OTI personnel 

13 Usability 

• The system must incorporate troubleshooting facilities that allow the support 
organization to identify and fix issues without engaging the engineering team 

• The process of converting applications to eStream sets and testing those eStream 
sets (certification) must be fairly expeditious (target an average no more than 1 
person's effort per application per day) 

1.4 Reliability 

• The internet-based software, software upgrade and application delivery system 
must be highly available to support ongoing remote installation and service work 

1.5 Performance 
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• The system must be no less than SOX more scalable than Windows Terminal 
Server/Citrix Metaframe when running client applications to make it a cost- 
effective high-volume client application delivery mechanism 

1.6 Scalability 

• Given sufficient hardware and network resources, the internet-based software, 
software upgrade and application delivery system must scale to handle any 
potential number of customers 

• The system can be easily expanded to support 64-bit file sizes 

• The system provides 1 28-bit application identifiers 

• The system supports applications that are composed of up to 2 billion files 

1.7 Security 

• It must be difficult enough to steal an application's code and data from the client 
that software providers are not concerned about using the system as a delivery 
mechanism 

• It must be very difficult to gain unauthorized entry to the internet-based software, 
software upgrade and application delivery system 

• It must be computationally infeasible to steal data or code while it is being 
distributed across the internet-based software, software upgrade and application 
delivery system 

• It must be computationally infeasible for the internet-based software, software 
upgrade and application delivery to be coerced by a third-party into delivering a 
Trojan 

• The internet-based software, software upgrade and application delivery must log 
all accesses so that an appropriate security audit can be periodically performed 

1.8 Portability 

• The server components must be portable across a wide variety of platforms and 
operating systems, including but not limited to: Windows NT 4, Windows 2000, 
Solaris UltraSPARC, HP-UX, and Linux 

1.9 Maintainability 

• The entire system must be component-wise upgradeable without requiring an 
interruption of service by any of our customers 

• The server components provide a web-based monitor and administrative console 
that can be used to diagnose and maintain an ASP site 

• The client components provide an interface that can be used to diagnose client 
problems and provide customer support 

• All application upgrades are backwards compatible 
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2.0 ASP Requirements 

The following requirements are primarily driven by an application service provider's 
business and operational needs. 

2.1 Functionality 

• The system can deliver client applications (e.g., productivity applications such as 
Microsoft Office, collaborative applications such as email and groupware, content 
creation applications such as Adobe Acrobat and Photoshop and client front end 
applications for client/server and 3-tier enterprise applications such as the People 
Soft client). Certification of additional client applications will require an 
additional certification fee. Omnishift does not guarantee that any given 
application will successfully certify even if a previous version of the application 
was certified 

• The following application eStream sets will be certified for the initial release of 
the product (the latest available versions should be assumed unless otherwise 
indicated): 



• Lotus 1-2-3 

• Lotus Word Pro 

• Microsoft Access 

• Microsoft Powerpoint 

• Microsoft Word 

• Microsoft Excel 

• Microsoft Outlook 

• Microsoft FrontPage 

• Adobe Photoshop 

• Adobe Premiere 

• Adobe Acrobat 

• Sonic Foundry Sound Forge 

• Macromedia Director 

• Macromedia Dreamweaver 

• Intuit Quicken 

• Intuit Turbotax 

• Qualcomm Eudora 



The system provides facilities for reporting application usage and usage patterns 
The system supports and reasonably enforces the following license payment 
models: 

• One-time charge for unlimited use 

• Pay per month 

The system supports and reasonably enforces the following license models: 

• Unlimited simultaneous use 

• No simultaneous use on multiple clients 
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• The system supports the ability to have the user view and agree to license terms 
upon subscription of a license 

• The system can deliver applications to any client capable of connecting to the 
internet over a standard HTTP connection 

• The system can service customers behind corporate and personal firewalls 

• The system can be integrated to work with any existing billing system and 
database, although a consulting and customization fee will be charged for 
integration with any database other than SQL server and any billing system other 
than? 

• The system provides an external API that makes it easy to interface with standard 
billing solutions 

• The system provides capabilities for reviewing billing information at the global, 
account and user levels 

• The system provides user management (create, edit, remove, enable, disable, 
remove, add to group, remove from group) capabilities including user groups 

• The system provides account management (create, edit, remove, enable, disable) 
capabilities that are integrated with the accounts in the billing system (a 1-to-l 
relationship between billing accounts or sub-accounts and system accounts) 

• The system provides application management (create, edit, enable, disable, 
remove, add to group, remove from group) capabilities including application 
groups 

• The system provides subscription and un-subscription capabilities that allow a 
user group access to an application group under a supported license model 

• The system provides capabilities for automatically upgrading an application upon 
next use, for allowing a client to optionally upgrade on each subsequent use or for 
informing a client that an upgrade is available for subscription on each subsequent 
use 

2.2 Localization 

• The server components and web interface components must be able to operate 
across the different languages supported by Windows, although only American 
English menus, panels, messages, help and documentation will be initially made 
available 

2.3 Usability 

• The system must not require any scheduled or unscheduled down time 

• The system provides appropriate help panels to the system administrator 

2.4 Reliability 

• Given sufficient hardware resources and an appropriate configuration, the system 
must be able to tolerate the failure of any one server either hosting the application 
or the software license management service without interrupting the service of 
any current or future clients 
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• The system should tolerate single failures in which a server or client transmits 
bogus messages or fails to transmit expected messages 

2.5 Performance 

• The system's overall price/performance makes it profitable to deliver low-cost 
client applications at volume levels 

• The system must be able to operate satisfactorily with an average bandwidth of 
256Kb/s or more 

• The system must be able to operate satisfactorily with an average latency of 0.25 
seconds or less 

• The system allows encryption to be enabled or disabled either globally or on a 
domain basis to improve system performance inside protected LAN environments 

• The system uses bandwidth discovery to determine whether compression is 
beneficial when sending massages between the client and the server 

2.6 Scalability 

• A fully-functional, non-redundant server system can be set up and configured to 
run on a single server machine 

• The system must be scalable, given sufficient machine resources, to support any 
number of clients and certified applications 

• The system can make effective use of the servers available to serve applications 
and license tokens without the need for constant administrative supervision and 
management 

2.7 Security 

• The server must be able to operate behind an industry-standard firewall solution 

• The system must withstand DOS (Denial Of Service) attacks with only localized 
degradations in service 

• The system must allow diagnostic modes to be locked out or disabled to ensure 
that no outside party, including OTI, can log on and administer the server 
components 

• The system allows for different levels of account, billing, user and subscription 
access so that the various capabilities can be allowed or denied on a user or user 
group basis 

• The system logs all application usage activity to a specific user and client so that 
appropriate security and usage audits can be performed 

2.8 Portability 

• The server hardware platform does not limit the platforms that applications can be 
delivered to from that server 

2.9 Maintainability 
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• The system must incorporate troubleshooting facilities to ease the burden of 
identifying, isolating and resolving operational issues 

• The server components must be able to log all abnormal activity and the 
administrator should be able to select which activity should be logged and which 
ignored 

• The server supports SNMP queries so that it can be integrated with a system 
management solution like Openview 

• The server must be easily configured so that alarms and alerts can be defined 
when important events occur 

• New applications and application upgrades can be installed without interrupting 
the current use of that application or any other applications by any client 

• It must be easy to add, remove, enable and disable application and license servers 
without affecting the rest of the system, requiring the system to be shut down or 
affecting application delivery to any current or future client (provided that 
sufficient alternative servers are available to maintain operations) 

3. 0 Client User Requirements 

The following requirements are primarily driven by a client user needs and expectations 
on the system. 

3.1 Functionality 

• The system supports multiple applications running on the same client, including 
multiple instances of the same application (within reasonable limitations of the 
operating system, the client hardware and the applications) 

• The system allows a single client to be simultaneously connected to multiple 
application service providers 

• The system provides the capability to view and edit billing, account, subscription 
and user information from a connected client 

• The system supports user roaming so that a user can access their applications 
from any machine that has installed the client and can access an application 
service provider over the network 

• The system allows user data to be saved on the local machine 

• The system allows applications to print to any local and network printers 
accessible to the client machine 

• The system allows any set of applications to interact with each other via OLE 
automation, COM and signals regardless of whether the applications are installed 
locally or delivered by the system 

• The system can be deactivated and reactivated by the user on demand 

• The system can be set up to work through a proxy server 

3.2 Localization 

• The client components must be able to operate across the different languages 
supported by Windows, although only American English menus, panels, 
messages, help and documentation will be initially made available 
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33 Usability 



• The average interactive response time of an application delivered by the system 
will not exceed 110% that which would be experienced by the user were running 
a locally-installed version of the application 

• The system client components are easily downloaded and installed 

• The system client components are easily uninstalled 

• The system detects the presence of a local version of the application being 
installed and warns the user that their local version will become unavailable while 
the system is operational 

• The system allows a pre-existing local version of the application to become 
available when the system's version of the application is uninstalled 

• The system does not require a reboot of the client whenever a new application is 
installed or uninstalled 

• The system removes all application-related traces from the client system when an 
application is uninstalled 

• The system must be able install itself and operate on a client with only 1 6MB of 
disk space available, although the system may request that the user free up and 
reserve more disk space to improve the performance of the system 

• The user can, through an advanced options menu, manage the size of the local 
system cache if they choose to override the system default behavior 

• The system provides the user access to information regarding cache usage, 
bandwidth usage and connectivity status 

• The user must be able to control which license to use when it can obtain a license 
from more than one connected application service provider 

• The system provides appropriate help panels to the user 

• The system allows a user to indicate that they do not wish to be informed of an 
optional upgrade again for a particular application and respects their choice on 
that matter 

• The system quickly releases non-simultaneous use licenses after their use so that 
the same user can use the license from another client 

• The system release non-simultaneous use licenses within a short span of time after 
a client that was using such a license crashes or is unexpectedly shut down 

• A client is given sufficient notification of an impending license expiration to save 
their work and take appropriate action to renew or extend the license 

• Upon an expiration of a license, the application is halted and the user given the 
ability to renew the license. The application is never terminated by the client 
components unless the user gives their consent to an actual termination 

3.4 Reliability 

• The client components can run indefinitely as they consume only a managed 
amount of system resources 

• The system must deliver code and data to the client identical to that it would have 
were the application residing locally 

• The client component can "heal" itself from deleted client dlls 
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• The system detects and recovers from garbled network messages although at 
potentially reduced performance 

3.5 Performance 

• The client must make a run/no run license decision quickly for any application 
execution request 

• The system collects profile information locally so that future accesses are tailored 
to local usage patterns 

3.6 Scalability 

• There must be no system-imposed limits on how many applications can be 
subscribed, installed or executed simultaneously 

3.7 Security 

• All subscription, financial and credit card transactions are performed securely 
making it computationally infeasible for a third party to obtain any such 
information 

• It must be computationally infeasible for the system to be coerced by a third-party 
into delivering a Trojan 

• It must be computationally infeasible for a third party to determine which 
application a client is executing 

• You must be able to specify that you do not wish your application usage 
information to be made available to any third party 

3.8 Portability 

• The system automatically delivers the appropriate version of a subscribed and 
installed application to the client provided that an appropriate certified version of 
the application is available at the application service provider's site for the client 
hardware and operating system configuration 

3.9 Maintainability 

• The client components should not require any user administration 

• The system client components should be upgradeable without requiring a re- 
installation or re-subscription of existing applications 
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readme.txt arai 



readme.txt 



This directory /docs is for estream design documentation only. 

This directory is reserved for high level documents such as the 
estream high level requirements and design documents. 

The subdirectories "client," "server," and "builder" are for documents 
specific to those pieces of estream. These directories will be used 
to contain component-wide documents, such as the server component 
framework and the estream set format document. Each directory will 
contain subdirectories for each sub-component. The names of the 
subcomponent directories will be the abbreviation or acronym for the 
component, in all lower case. Some client components will be ecm 
(cache manager), efsd (file system driver), epf (prefetching and 
fetching), and cni (client network interface). 

Another subdirectory of "documents" will be "eng". This will 
include all company-wide engineering documents, such as coding 
guidelines and design document templates. 

Document and directory names will contain no spaces. (These 
interoperate poorly with non-Microsoft platforms.) Files will contain 
the the name of the component followed by the acronym for the document 
type, separated by a hyphen Recognized document types are 

currently: 

LLD - Low level design 
REQ - requirements 
HLD - High level design 
SM - strawman 

Thus some documents would be 

CacheManger-LLD.doc (low level design for the cache manager) 
PrefetcherFetcher-SM.doc (prof iler/pref etcher component strawman) 
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Estream 1.0 Planning Document 
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Install Monitor 


Sanjay 


Done 


Done 


3 wk 


Builder GUI 


Sanjay 


Done 


Done 


1 wk 


FSRFD (Drivers) 


Sanjay 


Done 


Done 


2wk 


ApplnstallBlk structure 


David 


Done 


Not needed 




Profiler 


David 


Done 


Done 


2wk 


File Access Monitor 


David 


Done 


Done 


1 wk 


Packager 


David 


Done 


Done 


1 wk 
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Bob 
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TBD 


Server Group 










Web Server 


Bhaven 


Done 


Done 


8 wk 


Monitor 


Mike 


Done 


Done 


4 wk 


SLiM Server 


Amit 


Done 


Done 


2wk 


App Server 


Sameer 


Done 


Done 


4wk 


Admin Ul 


Bhaven 


TBD 


TBD 


TBD 


End User Ul 


Bhaven 


TBD 


TBD 


TBD 


Common Server Components 


Mike 


Done 


Done 


3 wk 


Messaging 


Sameer 


Done 


Done 


3wk 


Threads Package 


Sameer 


No Document 




1 wk 


Security Design 


Igor/Amit 


Not Done 


Not Done 


TBD 


Client Group 










Cache Prefetching 


Anne 


Done 


Done 


twk 


LSM + Plug in 


Anne 


Done 


Done 


1 wk 


Client Ul 


Anne 


Done 


Done 


1 wk 


Client Installer 


Anne 


Done 


Done 


1 wk 


Start Client 


Anne 


Done 


Done 


1 wk 


Application Install Mgr 


Nick 


Done 


Done 


TBD 


Piracy 


Nick 


Done 


Done 


TBD 


File Spoofer 


Curt 


Done 


Done 


1 wk 


eStream File System 


Curt 


Done 


Done 


8 wk 


NoCluster Driver 


Curt 


Status TBD 


Starus TBD 


2 days 


eStream Cache Manager 


Dan 


Done 


Done 


8 wk 


Client Network Interface 


Dan 


Done 


Done 


2wk 



Test 
Completed 



Implementation Plan 
Milestones 



i 




ECM (RAM disk cache) and EFSD executes a local "himom" executable 

Photoshop is installed locally and successfully executed from estream sets and applnstallblk produced by builder 
App Server and EMS integrated to copy -himom" executable using a dummy client 
App Server, EMS and CNI integrated to copy "himom" executable from "himom" estream sets 
office is installed locally and successfully executed from estream sets and applnstallblk produced by builder 
App Server, EMS, ENI, ECM and EFSD integrated to run "himom" from estream sets on server 
Following applications built and tested with local installation 
Adobe Premier 

Macromedia Director and Shockwave 
Corel Suite 
Lotus Suite 

Photoshop is installed by AIM and executed from estream sets on App server 

No Subscription 

No License Management 

RAM cache for ECM 

Installation of Photoshop using AIM 
Photoshop is installed by AIM and executed from estream sets on App server 

No Siim Server 

Disk based cache for ECM 





Estream includes initial prefetched pages and these pages are prefetched during installation 

Fully functional estream bits (includes initial prefetched pages) 

Client software is run as a service 

App Server is started by Monitor 

Admin Ul to stop and start app Server 

Application subscription from web server 

installation on client after subscription 
Testing environment is setup (configuration of 3 servers and one client) 
Photoshop runs with the following additional functionality 

Leads for milestone: Amit and Nick 

Slim Server 

http protocol 

CNl supports uique message ids for NAD 
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Functionality 

The ApplnstallBlock is a block of code and data associated with a particular application. 
This ApplnstallBlock contains the information needed to by the eStream client to 'initial- 
ize' the client machine before the eStream application is used for the first time. It also 
contains optional profiling data for increasing the runtime performance of that eStream 
application. 

The ApplnstallBlock is created offline by the eStream Builder program. First of all, the 
Builder monitors the installation process of a local version of the application installation 
program and records changes to the system. This includes any environment variables 
added or removed from the system, and any files added or modified in the system directo- 
ries. Files added to the application specific directory is not recorded in the Applnstall- 
Block to reduce the amount of time needed to send the ApplnstallBlock to the eStream 
client. Secondly, the Builder profiles the application to obtain the list of critical pages 
needed to run the application initially and an initial page reference sequence of the pages 
accessed during a sample run of the application. The ApplnstallBlock contains an op- 
tional application-specific initialization code. This code is needed when the default ini- 
tialization procedure is insufficient to setup the local machine environment for that par- 
ticular application. 

The ApplnstallBlock and the runtime data are packaged into the eStream Set by the 
Builder and then uploaded to the application server. After the eStream client subscribed 
to an application and before the application is run for the first time, the ApplnstallBlock 
is send by the server to the client. The eStream client invokes the default initialization 
procedure and the optional application-specific initialization code. Together, the default 
and the application-specific initialization procedure process the data in the Applnstall- 
Block to make the machine ready for eStreaming that particular application. 

Data type definitions 

The ApplnstallBlock is divided into the following sections: header section, variable sec- 
tion, file section, profile section, prefetch section, comment section, and code section. 
The header section contains general information about the ApplnstallBlock. The infor- 
mation includes the total byte size and an index table containing size and offset into other 
sections. In Windows version, the variable section consists of two registry tree structures 
to specify the registry entries added or removed from the OS environment. The file sec- 
tion is a tree structure consisting of the files copied to C drive during the application in- 
stallation. The profile section contains the initial set of block reference sequences during 
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Builder profiling of the application. The prefetch section consists of a subset of profiled 
blocks used by the Builder as a hint to the eStream client to prefetch initially. The com- 
ment section is used to inform the eStream client user of any relevant information about 
the application installation. Finally, the code section contains an optional program tai- 
lored for any application-specific installation not covered by the default eStream applica- 
tion installation procedure. In Windows version, the code section contains a Windows 
DLL. 

Here is a detailed description of each fields of the AppInstallBlock. 

Note: Little endian format is used for all the fields spanning more than 1 byte. Also, 
BlockNumber specifies blocks of 4K byte size. 

1. Header Section: 

The header section contains the basic information about that AppInstallBlock. This 
includes the versioning information, application identification, 

Core Header Structure: 

o Aib Version [4 bytes]: Magic number or appInstallBlock version number 
(which identifies the version of the appInstallBlock structure rather than the 
contents). 

o Appld [16 bytesj: this is an application identifier unique for each application. 
On Windows, this identifier is the GUID generated from the 'guidgen' pro- 
gram. Appld for Word on Win98 will be different from Word on WinNT if it 
tuins out that Word binaries are different between NT and 98. 

o VersionNo [4 bytes]: Version number. This allows us to inform the client that 
the appInstallBlock has changed for a particular appld. This is useful for 
changes to the AppInstallBlock due to minor patch upgrades in the applica- 
tion. 

o ClientOSBitMap [4 bytes]: Client OS supported bitmap or ID: for Win2K, 
Win98, WinNT and other future OSs we might support (it should be possible 
to say that this appInstallBlock is for more than one OS). 

o ClientOSServicePack [4 bytes]: We might want to store the service pack 
level of the OS for which this appInstallBlock has been created. Note that 
when this field is set we cannot use multiple OS bits in the above field Clien- 
tOSBitMap. 

o Flags (4 bytes]: Flags pertaining to AppInstallBlock 

■ Bit 0: Reboot - If set, the eStream client needs to reboot the 
machine after installing the AppInstallBlock on the client ma- 
chine. 

■ Bit 1 : Unicode - If set, the string characters are 2 bytes wide 
instead of 1 byte. 

o HeaderSize [2 bytes]: Total size in bytes of the header section, 
o Reserved [32 bytes]: Reserved spaces for future. 
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o NumberOfSections [1 byte]: Number of sections in the index table. This de- 
termines the number of entries in the index table structure described below: 

Index Table Structure: (variable number of entries) 

o SectionType [1 bytes]: The type of data describe in section. 0=file section, 
Invariable section, 2=prefetch section, 3=profile section, 4=comment section, 
5=code section. 

o SectionOffset {4 bytes]: The offset from the beginning of the file indicates 

the beginning of section, 
o Section Size [4 bytes]: The size in bytes of section. 

Variable Structure: 

o ApplicationNameLength [4 bytes]: Byte size of the application name 
o ApplicationName [X bytes] : Null terminating name of the application 

2. File Section: 

The file section contains a subset of the list of files needed by the application to run 
properly. This section does not enumerate files located in the standard application 
program directory. It consists of information about files copied into 'unusual' direc- 
tory during the installation of an application. If the file content is small, the file is 
copied to the client machine. Otherwise, the file is relocated to the standard program 
directory suitable for streaming. The file section data is list of trees stored in a con- 
tiguous sequence of address space according to the pre-order traversal of the trees. A 
node in the tree can correspond to one or more levels of directory. A parent-child 
node pair is combined into a single node if the parent node has only a single child. 
Parsing the tree from the root of the tree to a leaf node results in a fully legal Win- 
dows pathname including the drive letter. Each entry of the node in the tree consists 
of the following structure: 

Directory Structure: (variable number of entries) 

o Flags [4 byte]: Bit 0 is set if this entry is a directory 

o NumberOfChildren [2 bytes]: Number of nodes in this directory 

o DirectoryNameLength [4 bytes]: Length of the directory name 

o DirectoryName [X bytes]: Null terminating directory name 

Leaf Structure: (variable number of entries) 

o Flags [4 byte]: Bit 1 is set to 1 if this entry is a spoof or copied file name 
o FileVersion [4? bytes]: Version of the file GetFileVersionInfo() if the file is 
Win32 file image. Need variable file version size returned by GetFileVersion- 
InfoSize(). Otherwise use GetFileTime() to retrieve the file creation time, 
o FileNameLength [4 bytes]: Byte size of the file name 
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o FileName [X bytes]: Null terminating file name 

o DataLength [4 bytes): Byte size of the data. If spoof file, then data is the 

string of the spoof directory. If copied file, then data is the content of the file 
o Data [X bytes]: Either the spoof file name or the content of the copied file 

3. Add Variable and Remove Variable Sections: 

The add and remove variable sections contain the system variable changes needed to 
run the application. In Windows system, each section consists of several number of 
registry subtrees. Each tree is stored in a contiguous sequence of address space ac- 
cording to the pre-order traversal of the tree. A node in the tree can correspond to one 
or more levels of directory in the registry. A parent-child node pair is combined into 
a single node if the parent node has only a single child. Parsing the tree from the root 
of the tree to a leaf node results in a fully legal key name. The order of the trees is 
shown here. 

a. Registry Subsection: 

1. "HKCR": HKEYCLASSESROOT 

2. "HKCU": HKEY CURRENT USER 

3. "HKLM": HKEY LOCAL MACHINE 

4. "HKIF: HKEYUSERS 

5. "HKCC": HKEY CURRENT CONFIG 

Tree Structure: (5 entries) 

o ExistFlag (1 byte): Set to 1 if this tree exist, 0 otherwise, 
o Key or Value Structure entries [X bytesj: Serialization of the tree into 
variable number key or value structures described below. 

Key Structure: (variable number of entries) 

o Key Flag [1 byte]: Set to 1 if this entry is a key or 0 if it's a value structure 
o NumberOfSubchild [4 bytes]: Number of subkeys and values in this key 
directory 

o KeyNameLength (4 bytes]: Byte size of the key name 
o KeyName [X bytes]: Null terminating key name 

Value Structure: (variable number of entries) 

o KeyFlag (1 byte]: Set to 1 if this entry is a key or 0 if it's a value structure 
o ValueType [4 byte]: Type of values from the Win32 API function 

RegQueryValueExO: REG_SZ, REGBINARY, REGJDWORD, 

REG LINK, REG NONE, etc ... 
o ValueNameLength (4 bytes]: Byte size of the value name 
o ValueName [X bytes]: Null terminating value name 
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o ValueDataLengtb [4 bytes]: Byte size of the value data 
o ValueData [X bytes]: Value of the Data 

In addition to registry changes, an installation in Windows system may involve 
changes to the ini files. The following structure is used to communicate the ini file 
changes needed to be done on the eStream client machine. The ini entries are ap- 
pended to the end of the variable section after the 5 registry trees are enumerated. 

b. INI Subsection: (not supported in eStream 1.0) 

o NumFiles [4 bytes] : Number of INI files modified. 

File Structure: (variable number of entries) 

o FileNameLength [4 bytes]: Byte length of the file name 

o FileName [X bytes]: Name of the INI file 

o NumSection [4 bytes]: Number of sections with the changes 

Section Structure: (variable number of entries) 

o SectionNameLength [4 bytes]: Byte length of the section name 
o SectionName (X bytes]: Section name of an INI file 
o NumValues [4 bytes]: Number of values in this section 

Value Structure: (variable number of entries) 

o ValueLength [4 bytes]: Byte length of the value data 
o ValueData [X bytes]: Content of the value data 

4. Prefetch Section: 

The prefetch section contains a list of file blocks. The Builder profiler determines the 
set of file blocks critical for the initial run of the application. This data includes the 
code to start and terminate the application. It includes the file blocks containing for 
frequently used commands. For example, opening and saving of documents are fre- 
quently used commands and should be prefetched if possible. Another type of blocks 
to include in the prefetch section is the blocks associated with frequently accessed di- 
rectories and file metadata in this directory. The prefetch section is divided into two 
subsections. One part contains the critical blocks that are used during startup of the 
streamed application. The second part consists of the blocks accessed for common 
user operations like opening and saving of document. The format of the data is de- 
scribed below: 

a. Critical Block Subsection: 

o NumCriticalBlocks [4 bytes] : Number of critical blocks. 
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Block Structure: (variable number of entries) 

o FileNumber [4 bytes]: File Number of the file containing the block to pre- 
fetch 

o BIockNumber [4 bytes]: Block Number of the file block to prefetch 
b. Common Block Subsection: 

o NumCommonBlocks [4 bytes]: Number of critical blocks. 
Block Structure: (variable number of entries) 

o FileNumber [4 bytes]: File Number of the file containing the block to pre- 
fetch 

o BIockNumber [4 bytes]: Block Number of the file block to prefetch 

5* Profile Section: (not used in eStream 1.0) 

The profile section consists of a reference sequence of file blocks accessed by the ap- 
plication at runtime. Conceptually, the profile data is a two dimensional matrix. 
Each entry [row, column] of the matrix is the frequency a block row is followed by a 
block column. In any realistic applications of fair size, this matrix is very large and 
sparse. Proper data structure must be selected to store this sparse matrix efficiently in 
required storage space and minimize the overhead in accessing this data structure ac- 
cess. 

The section is constructed from two basic structures: row and column structures. 
Each row structure is followed by N column structures specified in the NumberCol- 
umns field. Note that this is an optional section. But with appropriate profile data, 
the eStream client prefetcher performance can be increased. 

Row Structure: (variable number of entries) 

o FileNumber [4 bytes]: File Number of the row block 
o BIockNumber [4 bytes]: Block Number of the row block 
o NumberCoIumns [4 bytes]: number of blocks that follows this block. This 
field determines the number of column structures following this field. 

Column Structure: (variable number of entries) 

o FileNumber [4 bytes]: File Number of the column block 

o BIockNumber [4 bytes]: Block Number of the column block 

o Frequency [4 bytes]: frequency the row block is followed by column block 
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6. Comment Section: 

The comment section is used by the Builder to describe this AppInstallBlock in more 
detail. 

o CommentLength [4 bytes]: Byte size of the comment string 
o Comment [X bytesj: Null terminating comment string 

7. Code Section: 

The code section consists of the application-specific initialization code needed to run 
on the eStream client to setup the client machine for this particular application. This 
section may be empty if the default initialization procedure in the eStream client is 
able to setup the client machine without requiring any application-specific instruc- 
tions. On the Windows system, the code is a DLL file containing two exported func- 
tion calls: Install®, UninstallQ. The eStream client loads the DLL and invokes the 
appropriate function calls. 

o CodeLength [4 bytesj: Byte size of the code 

o Code [X bytes]: Binary file containing the application-specific initialization 
code. On Windows, this is just a DLL file. 

8. LicenseAgreement Section: 

The Builder creates the license agreement section. The eStream client displays the li- 
cense agreement text to the end-user before the application is started for the first time. 
The end-user must agree to all licensing agreement set by the software vendor in or- 
der to use the application. 

o LicenseTextLength (4 bytes]: Byte size of the license text 

o LicenseAgreement [X bytesj: Null terminating license agreement string 



Open Issues 

o What is the size of the AppInstallBlock for a typical application like Office? 
o How large should the prefetch sections be for optimal run of an application? 

At minimum, it should contain at least start/termination code, 
o How should the AppInstallBlock handle application license agreement text 

string? Add a new section or use comment section. Does the dialog need to 

have exactly the same interface as the license agreement dialog on the local 

installation? 

o Currently, file section stores complete pathname including the drive letter. 
The installation may place files according to some variables like %System- 
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Root% or %UserProfile%. How does the Builder detect this so it can propa- 
gate this information to the client? 

Format of AppInstallBlock (part 1 of 2) 



12 



I I I 


i i i i i i i i i i i 


AIB Version 


AppID 
(GUID) 


AppID 
cont. 


Version No 


Client OS bitmap 


Client OS ServicePack 


Header 
Byte Size 


Number 
Sections 


Section 1 Type 
(Files) 


Section 1 Byte Offset 


Section 1 Byte Size 


Section 2 Type 
(Registry) 


Section 2 Byte Offset 


Section 2 Byte Size 


Section 3 Type 
(Prefetch) 


Section 3 Byte Offset 


Section 3 Byte Size 


Section 4 Type 
(Profile) 


Section 4 Byte Offset 


Section 4 Byte Size 


Section 5 Type 
(Comments) 


Section 5 Byte Offset 


Section 5 Byte Size 


Section 6 Type 
(Code) 


Section 6 Byte Offset 


Section 6 Byte Size 


App 
Name 
Length 


Application 
Name (x 
bytes) 



16 bytes 



Header 



Section 1 


I | I 


I I I 


I 




Offset 


Flags 


NumChildren 


Directory 
Name 
Length 


Directory Name (x bytes) 




Flags 


File Version 


File 
Name 
Length 


Data Length 


File Name (x 
bytes) 




File Name (x bytes) 
cont. 


Data Content (x bytes) 



\7 

x bytes 



File 



\7 



Section 2 
Offset 



Exist 
Flag 


4 

I I I 


8 x byte 
I I I 


Add/ 
Remove 
Reg 

n 


Key 
Flag 


Number 
children 


Value 
Type 


Key Name 
Length 


Key Name 
(x bytes) 


Value Data 
(x bytes) 


Key 
Flag 


Value 
Type 


Value Name 
Length 


Value Name (x bytes) 


Value Data (x bytes) 



Omnishift Technologies, Inc. 



8 



Company Confidential 



eStream AppInstallBlock Low Level Design 
Format of AppInstallBlock (part 2 of 2) 
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Converting Apps for Stream Delivery and subsequent Execution 



The Streamed Application Set Builder is a software program. It is used to convert locally 
installable applications into a data set suitable for streaming over the network. The 
streaming-enabled data set is called the Streamed Application Set (SAS). This document 
describes the procedure used to convert locally installable applications into the SAS. 

The application conversion procedure into the SAS consists of the several phases. In the 
first phase, the Builder program monitors the installation process of a local installation of 
the desired application for conversion. The Builder monitors any changes to the system 
and records those changes in an intermediate data structure. After the application is 
installed locally, the Builder enters the second phase of the conversion. In the second 
phase, the Builder program invokes the installed application executable and obtains 
sequences of frequently accessed file blocks of this application. Both the Builder 
program and the client software use the sequence data to optimize the performance of the 
streaming process. Once the sequencing information is obtained, the Builder enters the 
final phase of the conversion. In this phase, the Builder gathers all data obtained from the 
first two phase and processes the data into the Streamed Application Set. 

In the next sections, detailed descriptions of the three phases of the Builder conversion 
process are described. The three phases consists of installation monitoring (IM), 
application profiling (AP), and finally SAS packaging (SP). In most cases, the 
conversion process is general and applicable to all type of system. In places where the 
conversion is OS dependent, the discussion is focused on Microsoft Windows 
environment. Issues on conversion procedure for other OS environment are described in 
later sections. 

Installation Monitoring (IM) 

In the first phase of the conversion process, the Builder Installation Monitor (IM) 
component invokes the application installation program that installs the application 
locally. The IM observes all changes to the local computer during the installation. The 
changes may involve one or more of the following: changes to system or environment 
variables; and modifications, addition, or deletion of one or more files. The IM records 
all changes to the variables and files in a data structure to be sent to the Builder's 
Streamed Application Packaging component. In the following paragraphs, detailed 
description of the Installation Monitor is described for Microsoft Windows environment. 

In Microsoft Windows system, the Installation Monitor (IM) component consists of a 
kernel-mode driver subcomponent and a user-mode subcomponent. The kernel-mode 
driver is hooked into the system registry and file system function interface calls. The 
hook into the registry function calls allows the IM to monitor system variable changes. 
The hook into the file system function calls enables the IM to observe file changes. 
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Installation Monitor Kernel-Mode subcomponent (IM-KM) 

The IM-KM subcomponent monitors two classes of information during an application 
installation: system registry modifications and file modifications. Different techniques are 
used for each of these classes. 

To monitor system registry modifications, the IM-KM component replaces all kernel- 
mode API calls in the System Service Table that write to the system registry with new 
functions defined in the IM-KM subcomponent. When an installation program calls one 
of the API functions to write to the registry, the IM-KM function is called instead, which 
logs the modification data (including registry key path, value name and value data) and 
then forwards the call to the actual operating system defined function. The modification 
data is made available to the IM-UM subcomponent through a mechanism described 
below in Installation Monitor User-Mode subcomponent (IM-UM). 

To monitor file modifications, a filter driver is attached to the file system's driver stack. 
Each time an installation program modifies a file on the system, a function is called in the 
IM-KM subcomponent, which logs the modification data (including file path and name) 
and makes it available to the IM-UM using a mechanism described below in Installation 
Monitor User-Mode subcomponent (IM-UM). 

The mechanisms used for monitoring registry modifications and file modifications will 
capture modifications made by any of the processes currently active on the computer 
system. While the installation program is running, other processes that, for example, 
operate the desktop and service network connections may be running and may also 
modify files or registry data during the installation. This data must be removed from the 
modification data to avoid inclusion of modifications that are not part of the application 
installation. The IM-KM uses process monitoring to perform this filtering. 

To do process monitoring, the IM-KM installs a process notification callback function 
that is called each time a process is created or destroyed by the operating system. Using 
this callback function, the operating system sends the created process ID as well as the 
process ID of the creator (or parent) process. The IM-KM uses this information, along 
with the process ID of the IM-UM, to create a list of all of the processes created during 
the application installation. The IM-KM uses the following algorithm to create this list: 

1. Before the installation program is launched by the IM-UM, the IM-UM passes its 
own process ID to the IM-KM. Since the IM-UM is launching the installation 
application, the IM-UM will be the ancestor (parent, grandparent etc) of any 
process (with one exception - the Installer Service described below) that modifies 
files or registry data as part of the application installation. 

2. When the installation is launched and begins the creating processes, the IM-KM 
process monitoring logic is notified by the operating system via the process 
notification callback function. 

3. If the creator (parent) process ID sent to the process notification callback function 
is already in the process list, the new process is included in the list. 

When an application on the system modifies either the registry or files, and the IM-KM 
monitoring logic captures the modification data, but before making it available to the IM- 
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UM, it first checks to see if the process that modified the registry or file is part of the 
process list. It is only made available to the IM-UM if it is in the process list. 

It is possible that a process that is not a process ancestor of the IM-UM will make 
changes to the system as a proxy for the installation application. Using interprocess 
communication, an installation program may request than an Installer Service make 
changes to the machine. In order for the IM-KM to capture changes made by the Installer 
Service, the process monitoring logic includes a simple rule that also includes any 
registry or file changes that have been made by a process with the same name as the 
Installer Service process. On Windows 2000, for example, the Installer Service is called 
"msi.exe". 

Installation Monitor User-Mode subcomponent (IM-UM) 

The IM kernel-mode (IM-KM) driver subcomponent is controlled by the user-mode 
subcomponent (IM-UM). The EM-UM sends messages to the IM-KM to start and stop 
the monitoring process via standard I/O control messages known as IOCTLs. The 
message that starts the IM-KM also passes in the process ID of the IM-UM to facilitate 
process monitoring described in the IM-KM description. 

When the installation program modifies the computer system, the IM-KM signals a 
named kernel event. The IM-UM listens for these events during the installation. When 
one of these events is signaled, the IM-UM calls the IM-KM using an IOCTL message. 
In response, the M-KM packages data describing the modification and sends it to the 
IM-UM. 

The IM-UM sorts this data and removes duplicates. Also, it parameterizes all local- 
system-specific registry keys, value names, and values. For example, an application will 
often store paths in the registry that allow it to find certain files at run-time. These path 
specification must be replaced with parameters that can be recognized by the client 
installation software. 

A user interface is provided for the IM-UM that allows an operator of the Builder to 
browse through the changes made to the machine and to edit the modification data before 
the data is packaged into an SAS. 

Once the installation of an application completed, the IM-UM forwards data structures 
representing the file and registry modifications to the Streamed Application Packager. 
See figure 1 for the control flow diagram of IM module. 

Monitoring Application Configuration 

Using the techniques described above for monitoring file modifications and monitoring 
registry modifications, the builder can also monitor a running application that is being 
configured for a particular working environment. The data acquired by the IM-UM can 
be used to duplicate the same configuration on multiple machines, making it unnecessary 
for each user to configure his/her own application installation. 
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An example of this is a client server application for which the client will be streamed to 
the client computer system. Common configuration modifications can be captured by the 
IM and packed into the SAS. When the application is streamed to the client machine, it is 
already configured to attach to the server and begin operation. 

Application Profiling (AP) 

In the second phase of the conversion process, the Builder's Application Profiler (AP) 
component invokes the application executable program that is installed during the first 
phase of the conversion process. Given a particular user input, the executable program 
file blocks are accessed in a particular sequence. And the purpose of the AP is to 
captures this sequence data associated with some user inputs. This data is useful in 
several ways. 

First of all, frequently used file blocks can be streamed to the client machine before other 
less used file blocks. A frequently used file blocks is cached locally on the client cache 
before the user starts using the streamed application for the first time. This has the effect 
of making the streamed application as responsive to the user as the locally installed 
application by hiding any long network latency and bandwidth problems. 

Secondly, the frequently accessed files can be reordered in the directory to allow faster 
lookup of the file information. This optimization is useful for directories with large 
number of files. When the client machine looks up a frequently used file in a directory, it 
finds this file early in the directory search. In an application run with many directory 
queries, the performance gain is significant. 

Finally, the association of a set of file blocks with a particular user input allows the client 
machine to request minimum amount of data needed to respond to that particular user 
command. The profile data association with a user command is sent from the server to 
the client machine in the AppInstallBlock during the 'preparation' of the client machine 
for streaming. When the user on a client machine invokes a particular command, the 
codes corresponding to this command is prefetched from the server. 

The Application Profiler (AP) is not as tied to the system as the Installation Monitor (IM) 
but there are still some OS dependent issues. In the Windows system, the AP still has 
two subcomponents: kernel-mode (AP-KM) subcomponent and the user-mode (AP-UM) 
subcomponent. The AP-UM invokes the converting application executable. Then AP- 
UM starts the AP-KM to track the sequences of file block accesses by the application. 
Finally when the application exits after the pre-specified amount of sequence data is 
gathered, the AP-UM retrieves the data from AP-KM and forwards the data to the 
Streamed Application Packager. See Figure 2 for the control flow diagram of the AP 
module. 

Streamed Application Set Packaging (SP) 

In the final phase of the conversion process, the Builder's Streamed Application Set 
Packager (SP) component processes the data structure from IM and AP to create a data 
set suitable for streaming over the network. This converted data set is called the 
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Streamed Application Set and is suitable for uploading to the Streamed Application 
Servers for subsequent downloading by the stream client. Figure 3 shows the control 
flow of the SP module. 

Each file included in a Streamed Application Set is assigned a file number that identifies 
it within the SAS. 

The Streamed Application Set consists of the three sets of data from the Streamed 
Application Server's perspective. The three types of data are the Concatenation 
Application File (CAF), the Size Offset File Table (SOFT), and the Root Versioning 
Table (RVT). 

The Concatenation Application File (CAF) consists of all the files and directories needed 
to stream to the client. The CAF can be further divided into two subsets: initialization 
data set and the runtime data set. 

The initialization data set is the first set of data to be streamed from the server to the 
client. This data set contains the information captured by IM and AP needed by the client 
to prepare the client machine for streaming this particular application. This initialization 
data set is also called the AppInstallBlock (AIB). In addition to the data captured by the 
IM and AP modules, the SP is also responsible for merging any new dynamic profile data 
gathered from the client and the server. This data is merged into the existing 
AppInstallBlock to optimize subsequent streaming of the application. With the list of 
files obtained by the IM during application installation, the SP module separates the list 
of files into regular streamed files and the spoof files. The spoof files consists of those 
files not installed into standard application directory. This includes files installed into 
system directories and user specific directories. The detailed format description of the 
AppInstallBlock is described in Appendix B. 

The second part of the CAF consists of the runtime data set. This is the rest of the data 
that is streamed to the client once the client machine is initialized for this particular 
application. The runtime data consists of all the regular application files and the 
directories containing information about those application files. Detailed format 
description of the runtime data in the CAF section is described in Appendix A. The SP 
appends every files recorded by IM into the CAF and generates all directories. Each 
directory contains list of file name, file number, and the metadata associated with the files 
in that particular directory. 

The SP is also responsible for generating the SOFT file. This is a table used to index into 
the CAF for determining the start and the end of a file. The server uses this information 
to quickly access the proper file within the directory for serving the proper file blocks to 
the client. 

Finally, the SP creates the RVT file. The Root Versioning Table contains a list of root 
file number and version number. This information is used to track minor application 
patches and upgrades. Each entry in the RVT corresponds to one patch level of the 
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application with a corresponding new root directory. The SP generates new parent 
directories when any single file in that subdirectory tree is changed from the patched 
upgrade. The RVT is uploaded to the server and requested by the client at appropriate 
time for the most updated version of the application by a simple comparison of the 
client's Streamed Application root file number with the RVT table located on the server 
once the client is granted access authorization to retrieve the data. Figure 4 shows the 
internal representation of a simple SAS before and after a new file is added to a new 
version of an application. 

Data Flow Description 

The following list describes the data that is passed from one component to another. The 
numbers corresponds to the numbering in the Data Flow diagram. 

Install Monitor 

1 . The full pathname of the installer program is query from the user of the Builder 
program and is sent to the Install Monitor. 

2. The Install Monitor (IM) user-mode sends a read request to the OS to spawn a 
new process for installing the application on the local machine. 

3. The OS loads the application installer program into memory and run the 
application installer program. OS returns the process ID to the IM. 

4. The application program is started by the IM-UM. 

5. The application installer program sends read request to the OS to read the content 
of the CD. 

6. The CD media data files are read from the CD. 

7. The files are written to the appropriate location on the local hard-drive. 

8. IM kernel-mode captures all file read/write requests and all registry read/write 
requests by the application installer program. 

9. IM user-mode program starts the IM kernel-mode program and sends the request 
to start capturing all relevant file and registry data. 

10. IM kernel-mode program sends the list of all file modifications, additions, and 
deletions; and all registry modifications, additions, and deletions to the IM user- 
mode program. 

1 1 . Inform the SAS Builder UI that the installation monitoring has completed and 
display the file and registry data in a graphical user interface. 

Application Profiler 

12. Builder UI invokes Application Profiling (AP) user-mode program by querying 
the user for the list of application executable names to be profiled. The AP user- 
mode also query the user for division of file blocks into sections corresponding to 
the commands invoked by the user of the application being profiled. 

13. Application Profiler user-mode invokes each application executable in succession 
by spawning each program in a new process. The OS loads the application 
executable into memory, run the application executable, and return the process ID 
to the Application Profiler. 
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14. During execution, the OS on behalf of the application send the request to the 
hard-drive controller to read the appropriate file blocks into memory as needed by 
the application. 

15. The hard-drive controller returns all file blocks requested by the OS. 

16. Every file accesses to load the application file blocks into memory are monitored 
by the Application Profiler (AP) kernel-mode program. 

17. The AP user-mode program informs the AP kernel-mode program to start 
monitoring relevant file accesses. 

18. Application Profiler kernel-mode returns the file access sequence and frequency 
information to the user-mode program. 

19. Application Profiler returns the processed profile information. This has two 
sections. The first section is used to identify frequency of files accessed. The 
second section is used to list the file blocks for prefetch to the client. The file 
blocks can be further categorized into subsections according to the commands 
invoked by the user of the application. 

SAS Packager 

20. The Streamed Application Packager receives files and registry changes from the 
Builder UI. It also receives the file access frequency and a list of file blocks from 
the Profiler. File numbers are assigned to each file. 

21 . The Streamed Application Packager reads all file data from the hard-drive that are 
copied there by the application installer. 

22. The Streamed Application Packager also reads the previous version of Streamed 
Application Set for support of minor patch upgrades. 

23. Finally, the new Streamed Application Set data is stored back to non-volatile 
storage. 

24. For new profile data gathered after the SAS has been created, the packager is 
invoked to update the AppInstallBlock in the SAS with the new profile 
information. 

Mapping of Data Flow to Streamed Application Set (SAS) 

Step 7: Data gathered from this step consist of the registry and file modification, addition, 
and deletion. This data is mapped to the AppInstallBlock's File Section, Add 
Registry Section, and Remove Registry Section. 

Step 8 & 19: File data are copied to the local hard-drive then concatenated into part of the 
CAF contents. Part of the data is identified as spoof or copied files and the file 
names and/or contents are added to the AppInstallBlock. 

Step 1 5 & 2 1 : Part of the data gathered by the Profiler or gathered dynamically by the 

client is used in the AppInstallBlock as a prefetch hint to the client. Another part 
of the data is used to generate a more efficient SAS Directory content by ordering 
the files according the usage frequency. 

Step 20: If the installation program was an upgrade, SAS Packager needs previous 

version of the Streamed Application Set data. Appropriate data from the previous 
version is combined with the new data to form the new Streamed Application Set. 

Format of Streamed Application Set 
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The format of the Streamed Application Set consists of 3 sections: Root Version Table 
(RVT), Size Offset File Table (SOFT), and Concatenation Application File (CAF). The 
RVT section lists all versions of the root file numbers available in a Streamed 
Application Set. The SOFT section consists of the pointers into the CAF section for 
every file in the CAF. The CAF section contains the concatenation of all the files 
associated with the streamed application. The CAF section is made up of regular 
application files, SAS directory files, AppInstallBlock, and icon files. Please see 
Appendix A for detailed information on the content of the SAS file. Figure 6 shows a 
typical layout of the SAS file. 

OS dependent format 

The format of the Streamed Application Set is designed to be as portable as possible 
across all OS platforms. At the highest level, the format of CAF, SOFT, and RVT that 
make up the format of Streamed Application Set are completely portable across any OS 
platforms. One piece of data structure that is OS dependent is located in the initialization 
data set called AppInstallBlock in the CAF. This data is dependent on the type of OS due 
to the differences in low-level system differences among different OS. For example, the 
Microsoft Windows contain system environment variables called the Registry. The 
Registry has a particular tree format not found in other operating systems like UNIX or 
MacOS. 

Another OS dependent piece of data is located in the SAS directory files in the CAF. The 
directory contains file metadata information specific to Windows files. For example on 
the UNIX platform, there does not exist a hidden flag. This platform specific information 
needs to be transmitted to the client to fool the streamed application into believing that 
the application data is located natively on the client machine with all the associated file 
metadata in tack. If SAS is to be used to support streaming of UNIX or MacOS 
applications, file metadata specific to those systems will need to be recorded in the SAS 
directory. 

Lastly, the format of the file names itself is OS dependent. Applications running on the 
Windows environment inherit the old MSDOS 8.3 file name format. To support this 
properly, the format of the SAS Directory file in CAF requires an additional 8.3 field to 
store this information. This field is not needed in other operating systems like UNIX or 
MacOS. 

Device driver versus file system paradigm 

The SAS client Prototype is implemented using the 'device driver' paradigm. One of the 
advantages of the device driver approach is that the caching of the sector blocks is 
simpler. In the device driver approach, the client cache manager only needs to track 
sector number in its cache. In comparison with the 'file system' paradigm, more complex 
data structure is required to track a subset of a file that is cached on a client machine. 
This makes 'device driver' paradigm easier to implement. 

On the other hand, there are many drawbacks to the 'device driver' paradigm. On the 
Windows system, the device driver approach has problem supporting large number of 
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applications. This is due to the limitation on the number of assignable drive letters 
available in a Windows system (26 letters); and the fact that each application needs to be 
located on its own device. Note that having multiple applications on a device is possible, 
but then the server needs to maintain exponential number of devices that support all 
possible combinations of applications. This is too costly to maintain on the server. 

Another problem with the device driver approach is that the device driver operates at the 
disk sector level. This is a much lower level than operating at the file level in the file 
system approach. The device driver does not know anything about files. Thus, the 
device driver cannot easily interact with the file level issues. For example, spoofing files 
and interacting with OS buffer cache is nearly impossible with device driver approach. 
... But both spoofing files and interacting with OS buffer cache is need to get higher 
performance. 

See figure 7 and 8 for the differences between two paradigms. 
Implementation in the Prototype 

The prototype has been implemented and tested successfully on the Windows and Linux 
distributed system. The prototype is implemented using the 'device driver' paradigm as 
described above. The exact procedure for streaming application data is described next. 

First of all, the prototype server is started on either the Windows-based or Linux-based 
system. The server creates a large local file mimicking large local disk images. Once the 
disk images are prepared, it listens to TCP/IP ports for any disk sector read or write 
requests. 

Secondly, the conversion process is done on a Windows system via semi-manual 
procedure. The server disk image is 'mounted' on the local Z drive by making the proper 
TCP/IP connection to the server. Then the application installation program is invoked 
and the application is installed into the Z drive. This writes the application files into the 
Z drive device driver, through the TCP/IP connection, and finally on to the server disk 
image. At the same time, a file and registry monitoring program records all registry and 
file changes. This data is stored as an initialization file to be invoked on the client to 
prepare the client machine for streaming. 

Finally, after the application files is stored on the server disk image, the client prototype 
is started. The client connects to the server and 'mount' the server disk image as a local 
Z drive. Then the initialization file is invoked which setup the local registry variables 
and copy system files into proper directories. Once the local machine is prepared for 
streaming that particular application, the user can start using the application. When the 
application is first started, the pages are not located in the local buffer cache. The OS 
makes sector request to the Streamed Application device driver that forwards the sector 
request to the Streamed Application Cache Manager. If the sector is located in the 
Streamed Application cache, then the data is returned immediately. If the data is not 
located in the Streamed Application cache, then the request forwarded to the network 
component that sends the message to the server. The server finds the proper sector data 
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and returns the data to the client. The client Streamed Application Cache Manager 
caches the new sector data and forwards the sector data to the Streamed Application 
device driver. The device driver returns the sector data to the OS. 

Implementation of SAS Builder 

The SAS Builder has been implemented on the Windows-based platform. A preliminary 
Streamed Application Set file has been created for real-world applications like Adobe 
Photoshop. A simple extractor program has been developed to extract the SAS data on a 
pristine machine without the application installed locally. Once the extractor program is 
run on the SAS, the application runs as if it was installed locally on that machine. This 
process verifies the correctness of the SAS Building process. 



10 



Appendix A: Format of Streamed Application Set (SAS) 



Functionality 

The streamed application set is a data set associated with an application suitable for 
streaming over the network. The SAS is generated by the SAS Builder program. This 
program converts locally installable applications into the SAS. This document describes 
the format of the SAS. 

Note: Fields greater than a single byte is stored in little-endian format. The Stream 
Application Set (SAS) file size is limited to 2 A 64 bytes. The files in the CAF section are 
laid out in the same order as its corresponding entries in the SOFT table. 

Data type definitions 

The format of the SAS consists of 4 sections: header, Root Version Table (RVT), Size 
Offset File Table (SOFT), and Concatenation Application File (CAF) sections. 

1. Header section 

o MagicNumber [4 bytes]: Magic number identifying the file content with 
the SAS. 

o ESSVersion [4 bytesj: Version number of the SAS file format. 

o AppID [16 bytes]: A unique application ID for this application. This 

field must match the AppID located in the AppInstallBlock. Window 

Guidgen API is used to create this identifier, 
o Flags [4 bytes]: Flags pertaining to SAS. 
o Reserved [32 bytes]: Reserved spaces for future. 

o RVToffset [8 bytes] : Byte offset into the start of the RVT section. 

o RVTsize [8 bytes] : Byte size of the RVT section. 

o SOFToffset [8 bytes]: Byte offset into the start of the SOFT section. 

o SOFTsize [8 bytes]: Byte size of the SOFT section. 

o CAFoffset [8 bytes]: Byte offset into the start of the CAF section. 

o CAFsize [8 bytes]: Byte size of the CAF section. 

o VendorNamelsAnsi [1 byte]: 0 if the vendor name is in Unicode format. 

1 if the vendor name is in ANSI format, 
o VendorNameLength [4 bytes]: Byte length of the vendor name, 
o VendorName [X bytes): Name of the software vendor who created this 

application. Le. "Microsoft". Null-terminated, 
o AppBaseNamelsAnsi [1 byte]: 0 if the vendor name is in Unicode 

format. 1 if the vendor name is in ANSI format, 
o AppBaseNameLength [4 bytes]: Byte length of the application base 

name. 
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o AppBaseName [X bytes]: Base name of the application. Le. "Word 

2000". Null-terminated, 
o MessagelsAnsi [1 byte]: 0 if the vendor name is in Unicode format. 1 if 

the vendor name is in ANSI format, 
o MessageLength [4 bytes]: Byte length of the message text, 
o Message |X bytes]: Message text. Null-terminated. 

Root Version Table (RVT) section 

The Root version entries are ordered in a decreasing value according to their file 
numbers. The Builder generates unique file numbers within each SAS in a 
monotonically increasing value. So larger root file number implies later versions 
of the same application. The latest root version is located at the top of the section 
to allow the SAS Server easy access to the data associated with the latest root 
version. 

o NumberEntries [4 bytes]: Number of patch versions contained in this 
SAS. The number indicates the number of entries in the Root Version 
Table (RVT). 

o 

Root Version structure: (variable number of entries) 

o VersionNumber [4 bytes]: Version number of the root directory. 

o FileNumber [4 bytes]: File number of the root directory. 

o VersionNamelsAnsi [1 byte]: 0 if the vendor name is in Unicode format. 

1 if the vendor name is in ANSI format, 
o VersionNameLength [4 bytes]: Byte length of the version name 
o VersionName {X bytes]: Application version name. I.e. "SP 1". 
o Metadata [32 bytes]: See SAS FS Directory for format of the metadata. 

Size Offset File Table (SOFT) section 

The SOFT table contains information to locate specific files in the CAF section. 
The entries are ordered according to the file number starting from 0 to 
NumberFiles-1 . The start of the SOFT table is aligned to 8 bytes boundary for 
faster access. 

SOFT entry structure: (variable number of entries) 

o Offset [8 bytes]: Byte offset into CAF of the start of this file, 
o Size [8 bytes]: Byte size of this file. The file is located from address 
Offset to Offset+Size. 

Concatenation Application File (CAF) section 

CAF is a concatenation of all file or directory data into a single data structure. 
Each piece of data can be a regular file, an AppInstallBlock, an SAS FS 
directory file, or an icon file. 
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a. Regular Files 



o FileData [X bytes]: Content of a regular file 

b. ApplnstallBlock (See ApplnstallBlock document for detail format) 

A simplified description of the ApplnstallBlock is listed here. For exact detail 
of the individual fields in the ApplnstallBlock, please see Appendix B. 

o Header section [X bytes]: Header for ApplnstallBlock containing 

information to identify this ApplnstallBlock. 
o Files section [X bytes]: Section containing file to be copied or spoofed, 
o AddVariable section [X bytes]: Section containing system variables to 

be added. 

o RemoveVariable section [X bytes]: Section containing system variables 
to be removed. 

o Prefetch section [X bytes]: Section containing pointers to files to be 

prefetched to the client, 
o Profile section [X bytes] : Section containing profile data, 
o Comment section [X bytes]: Section containing comments about 

ApplnstallBlock. 

o Code section [X bytes]: Section containing application-specific code 
needed to prepare local machine for streaming this application 

o LicenseAgreement section [X bytes]: Section containing licensing 
agreement message. 

c. SAS Directory 

An SAS Directory contains information about the subdirectories and files 
located within this directory. This information is used to store metadata 
information related to the files associated with the streamed application. This 
data is used to fool application into thinking that it is running locally on a 
machine when most of the data is resided elsewhere. 

The SAS directory contains information about files in its directory. The 
information includes file number, names, and metadata associated with the 
files. 

o MagicNumber [4 bytes]: Magic number for SAS directory file. 

o ParentFilelD [16+4 bytes]: AppID+FileNumber of the parent directory. 

AppID is set to 0 if the directory is the root, 
o SelfFilelD [16+4 bytes]: AppID+FileNumber of this directory, 
o NumFiles [4 bytes]: Number of files in the directory. 



Variable-Sized File Entry: 
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o UsedFlag [1 byte]: 1 for used, 0 for unused, 
o ShortLen (1 byte]: Length of short file name, 
o LongLen [2 byte]: Length of long file name. 

o NameHash [4 bytes]: Hash value of the short file name for quick lookup 
without comparing whole string. 

o ShortName [24 bytes]: 8.3 short file name in Unicode. Not null- 
terminated. 

o FilelD [16+4 bytes]: AppID+FileNumber of each file in this directory. 

o Metadata [32 bytes]: The metadata consists of file byte size (8 bytes), 
file creation time (8 bytes), file modified time (8 bytes), attribute flags 
(4 bytes), SAS flags (4 bytes). The bits of the attribute flags have the 
following meaning: 

■ Bit 0: Read-only - Set if file is read-only 

» Bit 1 : Hidden - Set if file is hidden from user 

■ Bit 2: Directory - Set if the file is an SAS Directory 

■ Bit 3: Archive - Set if the file is an archive 

■ Bit 4: Normal - Set if the file is normal 

■ Bit 5: System - Set if the file is a system file 

■ Bit 6: Temporary - Set if the file is temporary 
The bits of the SAS flags have the following meaning: 

■ Bit 0: ForceUpgrade - Used only on root file. Set if client is 
forced to upgrade to this particular version if the current root 
version on the client is older. 

■ Bit 1 : RequireAccessToken - Set if file require access token 
before client can read it. 

■ Bit 2: Read-only - Set if the file is read-only 

o LongName [X bytes]: Long filename in Unicode format with null- 
termination character. 

Icon files 

o IconFileData [X bytes]: Content of an icon file. 



14 



Appendix B: Format of AppInstallBIock 
Functionality 

The AppInstallBIock is a block of code and data associated with a particular application. 
This AppInstallBIock contains the information needed to by the SAS client to 'initialize' 
the client machine before the streamed application is used for the first time. It also 
contains optional profiling data for increasing the runtime performance of that streamed 
application. 

The AppInstallBIock is created offline by the SAS Builder program. First of all, the 
Builder monitors the installation process of a local version of the application installation 
program and records changes to the system. This includes any environment variables 
added or removed from the system, and any files added or modified in the system 
directories. Files added to the application specific directory is not recorded in the 
AppInstallBIock to reduce the amount of time needed to send the AppInstallBIock to the 
SAS client. Secondly, the Builder profiles the application to obtain the list of critical 
pages needed to run the application initially and an initial page reference sequence of the 
pages accessed during a sample run of the application. The AppInstallBIock contains an 
optional application-specific initialization code. This code is needed when the default 
initialization procedure is insufficient to setup the local machine environment for that 
particular application. 

The AppInstallBIock and the runtime data are packaged into the SAS by the Builder and 
then uploaded to the application server. After the SAS client subscribed to an application 
and before the application is run for the first time, the AppInstallBIock is send by the 
server to the client. The SAS client invokes the default initialization procedure and the 
optional application-specific initialization code. Together, the default and the 
application-specific initialization procedure process the data in the AppInstallBIock to 
make the machine ready for streaming that particular application. 

Data type definitions 

The AppInstallBIock is divided into the following sections: header section, variable 
section, file section, profile section, prefetch section, comment section, and code section. 
The header section contains general information about the AppInstallBIock. The 
information includes the total byte size and an index table containing size and offset into 
other sections. In Windows version, the variable section consists of two registry tree 
structures to specify the registry entries added or removed from the OS environment. 
The file section is a tree structure consisting of the files copied to C drive during the 
application installation. The profile section contains the initial set of block reference 
sequences during Builder profiling of the application. The prefetch section consists of a 
subset of profiled blocks used by the Builder as a hint to the SAS client to prefetch 
initially. The comment section is used to inform the SAS client user of any relevant 
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information about the application installation. Finally, the code section contains an 
optional program tailored for any application-specific installation not covered by the 
default streamed application installation procedure. In Windows version, the code 
section contains a Windows DLL. The following is a detailed description of each fields 
of the AppInstallBlock. 

Note: Little endian format is used for all the fields spanning more than 1 byte. Also, 
BlockNumber specifies blocks of 4K byte size. 

1. Header Section: 

The header section contains the basic information about this AppInstallBlock. This 
includes the versioning information, application identification, and index into other 
sections of the file. 

Core Header Structure: 

o AibVersion [4 bytes]: Magic number or appInstallBlock version number 
(which identifies the version of the appInstallBlock structure rather than the 
contents). 

o Appld [16 bytes]: this is an application identifier unique for each application. 
On Windows, this identifier is the GUK) generated from the 'guidgen' 
program. Appld for Word on Win98 will be different from Word on WinNT if 
it turns out that Word binaries are different between NT and 98. 

o VersionNo [4 bytes]: Version number. This allows us to inform the client that 
the appInstallBlock has changed for a particular appld. This is useful for 
changes to the AppInstallBlock due to minor patch upgrades in the 
application. 

o ClientOSBitMap [4 bytes]: Client OS supported bitmap or ID: for Win2K, 
Win98, WinNT and other future OSs we might support (it should be possible 
to say that this appInstallBlock is for more than one OS). 

o ClientOSServicePack [4 bytes]: We might want to store the service pack 
level of the OS for which this appInstallBlock has been created. Note that 
when this field is set we cannot use multiple OS bits in the above field 
ClientOSBitMap. 

o Flags [4 bytes]: Flags pertaining to AppInstallBlock 

■ Bit 0: Reboot - If set, the SAS client needs to reboot the machine after 
installing the AppInstallBlock on the client machine. 

■ Bit 1 : Unicode - If set, the string characters are 2 bytes wide instead of 1 
byte. 

o HeaderSize [2 bytes] : Total size in bytes of the header section, 
o Reserved [32 bytes]: Reserved spaces for future. 

o NumberOfSections [1 byte]: Number of sections in the index table. This 
determines the number of entries in the index table structure described below: 

Index Table Structure: (variable number of entries) 
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o SectionType [1 bytesj: The type of data describe in section. 0=file section, 
l=variable section, 2=prefetch section, 3=profile section, 4=comment section, 
5=code section. 

o SectionOffset [4 bytes]: The offset from the beginning of the file indicates 

the beginning of section, 
o SectionSize [4 bytes]: The size in bytes of section. 

Variable Structure: 

o AppIicationNamelsAnsi [1 byte]: 1 if ansi, 0 if Unicode. 

o ApplicationNameLength [4 bytes]: Byte size of the application name 

o AppIicationName [X bytes): Null terminating name of the application 

2. File Section: 

The file section contains a subset of the list of files needed by the application to run 
properly. This section does not enumerate files located in the standard application 
program directory. It consists of information about files copied into 'unusual' 
directory during the installation of an application. If the file content is small, the file 
is copied to the client machine. Otherwise, the file is relocated to the standard 
program directory suitable for streaming. The file section data is list of trees stored in 
a contiguous sequence of address space according to the pre-order traversal of the 
trees. A node in the tree can correspond to one or more levels of directory. A parent- 
child node pair is combined into a single node if the parent node has only a single 
child. Parsing the tree from the root of the tree to a leaf node results in a fully legal 
Windows pathname including the drive letter. Each entry of the node in the tree 
consists of the following structure: 

Directory Structure: (variable number of entries) 

o Flags [4 byte] : Bit 0 is set if this entry is a directory 

o NumberOfChildren [2 bytes]: Number of nodes in this directory 

o DirectoryNameLength |4 bytes]: Length of the directory name 

o DirectoryName (X bytes]: Null terminating directory name 

Leaf Structure: (variable number of entries) 

o Flags [4 byte]: Bit 1 is set to 1 if this entry is a spoof or copied file name 

o FileVersion [8 bytes]: Version of the file GetFileVersionInfo() if the file is 
Win32 file image. Need variable file version size returned by 
GetFileVersionInfoSize(). Otherwise use file size or file modified time to 
compare which file is the later version. 

o FileNameLength [4 bytes]: Byte size of the file name 

o FileName [X bytes]: Null terminating file name 
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o DataLength [4 bytes]: Byte size of the data. If spoof file, then data is the 

string of the spoof directory. If copied file, then data is the content of the file 
o Data [X bytes]: Either the spoof file name or the content of the copied file 

3. Add Variable and Remove Variable Sections: 

The add and remove variable sections contain the system variable changes needed to 
run the application. In Windows system, each section consists of several number of 
registry subtrees. Each tree is stored in a contiguous sequence of address space 
according to the pre-order traversal of the tree. A node in the tree can correspond to 
one or more levels of directory in the registry. A parent-child node pair is combined 
into a single node if the parent node has only a single child. Parsing the tree from the 
root of the tree to a leaf node results in a fully legal key name. The order of the trees 
is shown here. 

a. Registry Subsection: 

1 . "HKCR": HKEYCLAS SESROOT 

2. "HKCU": HKEYCURRENTUSER 

3. "HKLM": HKEY LOCAL MACHINE 

4. "HKUS": HKEYJJSERS 

5. "HKCC": HKEY CURRENT CONFIG 

Tree Structure: (5 entries) 

o ExistFlag [1 byte]: Set to 1 if this tree exist, 0 otherwise, 
o Key or Value Structure entries [X bytes]: Serialization of the tree into 
variable number key or value structures described below. 

Key Structure: (variable number of entries) 

o Key Flag [1 byte]: Set to 1 if this entry is a key or 0 if it's a value structure 
o NumberOfSubchild [4 bytes]: Number of subkeys and values in this key 
directory 

o KeyNameLength [4 bytes]: Byte size of the key name 
o KeyName [X bytes]: Null terminating key name 

Value Structure: (variable number of entries) 

o Key Flag [1 byte]: Set to 1 if this entry is a key or 0 if it's a value structure 
o ValueType [4 byte]: Type of values from the Win32 API function 

RegQueryValueEx(): REG_SZ, REGJBINARY, REGJDWORD, 

REGJJNK, REGNONE, etc. . . 
o ValueNameLength |4 bytes]: Byte size of the value name 
o ValueName [X bytes]: Null terminating value name 
o ValueDataLength [4 bytes]: Byte size of the value data 
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o ValueData [X bytes] : Value of the Data 



In addition to registry changes, an installation in Windows system may involve 
changes to the ini files. The following structure is used to communicate the ini file 
changes needed to be done on the SAS client machine. The ini entries are appended 
to the end of the variable section after the 5 registry trees are enumerated. 

b. INI Subsection: 

o NumFiles [4 bytes]: Number of INI files modified. 

File Structure: (variable number of entries) 

o FileNameLength [4 bytes]: Byte length of the file name 

o FileName [X bytes] : Name of the INI file 

o NumSection [4 bytes]: Number of sections with the changes 

Section Structure: (variable number of entries) 

o SectionNameLength [4 bytes]: Byte length of the section name 
o SectionName [X bytes]: Section name of an INI file 
o NumValues [4 bytes]: Number of values in this section 

Value Structure: (variable number of entries) 

o ValueLength [4 bytes]: Byte length of the value data 
o ValueData [X bytes]: Content of the value data 

4. Prefetch Section: 

The prefetch section contains a list of file blocks. The Builder profiler determines the 
set of file blocks critical for the initial run of the application. This data includes the 
code to start and terminate the application. It includes the file blocks containing for 
frequently used commands. For example, opening and saving of documents are 
frequently used commands and should be prefetched if possible. Another type of 
blocks to include in the prefetch section is the blocks associated with frequently 
accessed directories and file metadata in this directory. The prefetch section is 
divided into two subsections. One part contains the critical blocks that are used 
during startup of the streamed application. The second part consists of the blocks 
accessed for common user operations like opening and saving of document. The 
format of the data is described below: 

a. Critical Block Subsection: 

o NumCriticalBlocks [4 bytes]: Number of critical blocks. 
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Block Structure: (variable number of entries) 

o FileNumber |4 bytes]: File Number of the file containing the block to 
prefetch 

o BlockNumber [4 bytes]: Block Number of the file block to prefetch 
b. Common Block Subsection: 

o NumCommonBIocks [4 bytes]: Number of critical blocks. 
Block Structure: (variable number of entries) 

o FileNumber [4 bytes): File Number of the file containing the block to 
prefetch 

o BlockNumber [4 bytes]: Block Number of the file block to prefetch 

5. Profile Section: 

The profile section consists of a reference sequence of file blocks accessed by the 
application at runtime. Conceptually, the profile data is a two dimensional matrix. 
Each entry [row, column] of the matrix is the frequency a block row is followed by 
block column. In any realistic applications of fair size, this matrix is very large and 
sparse. Proper data structure must be selected to store this sparse matrix efficiently 
required storage space and minimize the overhead in accessing this data structure 
access. 

The section is constructed from two basic structures: row and column structures. 
Each row structure is followed by N column structures specified in the 
NumberColumns field. Note that this is an optional section. But with appropriate 
profile data, the SAS client prefetcher performance can be increased. 

Row Structure: (variable number of entries) 

o FileNumber [4 bytes]: File Number of the row block 
o BlockNumber [4 bytes]: Block Number of the row block 
o NumberColumns [4 bytes]: number of blocks that follows this block. This 
field determines the number of column structures following this field. 

Column Structure: (variable number of entries) 

o FileNumber [4 bytes]: File Number of the column block 

o BlockNumber [4 bytesj: Block Number of the column block 

o Frequency [4 bytes]: frequency the row block is followed by column block 
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6. Comment Section: 

The comment section is used by the Builder to describe this AppInstallBlock in more 
detail 

o CommentLengthlsAnsi [1 byte]: 1 if string is ansi, 0 if Unicode format, 
o CommentLength [4 bytes]: Byte size of the comment string 
o Comment [X bytes]: Null terminating comment string 

7. Code Section: 

The code section consists of the application-specific initialization code needed to run 
on the SAS client to setup the client machine for this particular application. This 
section may be empty if the default initialization procedure in the SAS client is able 
to setup the client machine without requiring any application-specific instructions. 
On the Windows system, the code is a DLL file containing two exported function 
calls: InstallQ, UninstallQ. The SAS client loads the DLL and invokes the 
appropriate function calls. 

o CodeLength [4 bytes]: Byte size of the code 

o Code [X bytes]: Binary file containing the application-specific initialization 
code. On Windows, this is just a DLL file. 

8. LicenseAgreement Section: 

The Builder creates the license agreement section. The SAS client displays the 
license agreement text to the end-user before the application is started for the first 
time. The end-user must agree to all licensing agreement set by the software vendor 
in order to use the application. 

o LicenseTextlsAnsi [1 byte]: 1 if ansi, 0 if Unicode format, 
o LicenseTextLength [4 bytes]: Byte size of the license text 
o LicenseAgreement [X bytes]: Null terminating license agreement string 
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Figure 1: Builder Install Monitor (IM) Control Flow Diagram 
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Figure 2: Builder Application Profiler (AP) Control Flow Diagram 
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Figure 3: Builder SAS Packager (SP) Control Flow Diagram 
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Figure 4: Internal Representation of SAS with Support 

for Versioning 
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Figure 6: Content of a Streamed Application Set 
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Figure 10: Format of ApplnstallBlock (part 1 of 2) 
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This document contains the high level design of the eStream Application Builder. The 
Builder is used to "prepare" an application before it can be eStreamed. This document 
describes the high level design of the application installation monitoring, file relocation 
and mapping, gathering of the initial profiling information of an application, the 
packaging of the eStream Set, and the merging of the newly uploaded user profile data. 

Note: all references to "user" should be understood to mean the user of the Builder (i.e. 
the person who is responsible for creating eStream sets) and not the end-user of eStream 
technology. 

This document described these steps involved in the preparation of the application: 
Installation Monitoring, Application Profiling, and eStream Packaging. 

Modules 

Installation Monitor: 

• When the application is installed, we need to monitor the installation to see 
various "things" taking place on the computer. These could be: 

■ Various updates to the System Registry 

■ Files added to the Install directories (i.e. directories where application bits are 
copied as specified by the installing user). Lets call this group F|. 

■ Files added/updated to the Shared directories (e.g. "Program Files\Common 
Files"). Lets call this group Fo 

» Files added/updated to the System directories (e.g. "WinNT\System32"). Lets 
call this group Fs. 

■ Files added/updated to the User specific directories (e.g. "Documents and 
Settings\spujare\Application Data"). Lets call this group Fy. 

Note that once this information is gathered by the "Installation Monitor", a single 
"Installation Set" is prepared where all the files are stored in a single directory 
hierarchy. Note that files in the Fc, Fs and Fu groups (i.e. Fcsu group) are also 
stored here. For these files a "mapped location" is created under the single 
directory hierarchy. The Installation Set typically creates a map of all files (called 
ISM for Installation Set Map) described above with each entry containing the 
following info: 

1. fileld for the file 

2. location and name of the file. Note that the location will be the actual location 
for Fj files, but mapped location for the Fcsu files. 



• After we gather the above information, we need to prepare a 'Tile Relocation 
Map" (FRM) that is used by the client file spoofer to spoof references to any file 
in the common file group (i.e. Fcsu)- For example: when the eStreamed app 
makes a reference to a file C : \Program Files\Word\Foobar, the file 
spoofer actually redirects that reference to Z : \ Program 
Files\Word\Foobar. It does that because of the File Relocation Map. Each 
entry in the FRM typically has the following info: 

1 . fileld (which references an entry in the ISM). 

2. Actual location where the application expects it (i.e. C : \Program 
Files\Word\Foobar) . 

Profile Module: 

During the application building process, the Builder program queries the user for the 
name of the application executable. Then Builder program starts and terminates the 
application executable immediately to gather initial sequence of the application page 
access pattern. After the initial seed of profile data is acquired, the Profile Sequence 
Matrix is combined with other appInstallBlock data gathered from the Install Monitor. 

Profile Sequence Matrix is a 2D matrix of a profile data. Each entry of the matrix 
[column C, row R] is an integer value indicating the number of times a page R is 
requested following the request of page C. This successor request pattern is the page 
requests missed in the eStream cache manager. 

Package Module: 

In the final phase of the Builder program, the appInstallBlock is encapsulated into a 
special installation executable and the application files is archived into a single 
compressed package. The install executable containing the appInstallBlock and the 
archive of application files can then be placed in a suitable eStream Set server for ASP to 
download to their machines. 

Merge Module: (not supported in version 1.0) 

During normal eStream application usage, the eStream client gathers profile information 
for that particular run of the application. Then at the termination of an eStream 
application, it uploads the new Profile Sequence Matrix to the Profile Server. The clients 
should not upload the Profile Sequence Matrix from previous runs because the Profile 
Server has no mechanism for distinguishing between previously uploaded data and the 
newly acquired data. 

At appropriate time, the Builder is invoked to merge the newly uploaded per-user Profile 
Sequence Matrix into a collective Matrix. The merging algorithm may be designed with 
some heuristics to prevent the data biasing toward power users. This collective Matrix 
can be reinserted into the appropriate appInstallBlock then downloaded by any requesting 
eStream clients. 



Kernel Device Drivers: 



In addition, kernel device drivers are used to actually hook into the operating system to 
monitor the registry and file changes during installation of the application. This is 
accomplished by the FSRFD module. 

The kernel device driver is also used for gathering monitoring file block references from 
the operating system to the file system. This is accomplished by the File Access Monitor. 
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Interfaces 



The interfaces are divided into three use cases: application installation monitoring, 
application profiling, and merging of the uploaded profile data. 

Use Case #1 : Install Monitor 

Al . Builder UI to Install Monitor - send the name of the application installation 
executable 

A2. App CD to Install Monitor - the CD containing the application is fed into the 

installation monitor module 
A3. Install Monitor to Installation App - invoke the installation program 
A4. Installation App to FSRFD - monitor all changes to the registry and files when 

installation program write to local file system 
A5. FSRFD to Install Monitor - send all registry and file changes 
A6. Install Monitor to itself - repeat all applications in the suite and merge all data 
A7. Install Monitor to Manual Intervention - send a list of registry and file captured 

by the install monitor to UI and allow user to add or delete any entries 
A8. Manual Intervention to Package Manager - send the final registry and file 

relocation data to the packager 
A9. Package Manager to database - data set is packaged into appInstallBlock and the 

rest of the application files suitable for eStreaming 

Use Case #2: Profiling 

Bl . Builder UI to Profile Manager - send the name of the application executable 
B2. Profile Manager to Run App - invoke the application 

B3. Run App to eStream File Access Monitor - record sequences of page requests 
B4. File Access Monitor to Profile Manager - save the profile information 
B5. Profile Manager to Profile Manager - repeat for each application in the suite 
B6. Profile Manager to Package Manager - send all profile data for merging into a 
single data 

B7. database to Package Manager - get eStream Set from the database 
B8. Package Manager to database - save the updated eStream Set 

Use Case #3: Merging Profile data (not supported in version 1.0) 

CI . Builder UI to Merger - send the application name with profile data to merge 
C2. database to Merger - get uploaded Profile Sequence Matrix from the Profile 
Server 

C3. database to Merger - get the old appInstallBlock from database 

C4. Merger to Package Manager - reinsert the Profile data into appInstallBlock 

C5. Package Manager to database - save the updated appInstallBlock 

Requirements 

Please see eStreaml.O-REQ.doc for the most up-to-date list of the Builder requirements. 
This requirement list may not contain the most recent changes. Each requirement is 
identified by a tag such as R-XXXX for easy references elsewhere in the document. 



R-Background: The installation monitor runs in the background, when an 
eStream application is installed as part of its preparation or building. 
R-RegistryCapture: The installation monitor captures all the updates to the 
System Registry that take place during the install. These updates are captured as a 
.REG file. Note that registry key deletions are also captured and stored in the 
.REG file. Please see the LLD doc by Charles Booher about the Registry spoofing 
database. 

R-FileCapture: The installation monitor records all the files created in the two 
kinds of directories: the install directory (the Fj group described above) and the 
common directories (the Fcsu group). All the files created are copied to the 
Installation Set and the File Relocation Map (FRM) created for the Fcsu group 
files. <Note: as far as a system common DLL is concerned, the eStream client 
should (a) overwrite the existing DLL if it exists (b) spoof it if doesn't exist. This 
is necessary because some installations may depend on newer versions of, say 
MSVCRT.DLL and in Windows there is no way to maintain different versions of 
the same DLL>. 

R-InitialProfiling: The Builder must be able to gather initial set of application 
profile data. This data consists of the page access pattern for starting and 
immediately shutting down an application. 

R-Packaging: The Builder must package the eStream Set into a easily 
manageable packages suitable for ASP administrators to download to their 
servers. The package can be divided into two sets: 

1 . Installation Set - an appInstallBlock which is a set of data needed to setup the 
client machine for running a particular eStream application. The 
appInstallBlock is converted into an installation executable for simplifying the 
initial application set-up on the client machine. 

2. Run-time Set - a set of files associated with a particular application. At run- 
time, appropriate pages from this set of files is streamed to the client. 

R-Merging (not supported in version 1.0): The Builder must be able to collect 
per-user profile data from the Profile Server and merge the profile data into a 
combined data usable for updating the profile data in the appInstallBlock. This 
profile data can also be collected for use by the ASP or application developers. 
R-NoQuietOperation: The Builder is not required to be run in an environment 
where no other applications are running. But, since the Builder operates by 
invoking application installation program, it inherits any restrictive "Quiet- 
Operation" requirement from the installation program. Thus, if the installation 
program of an application has a "Quiet-Operation" requirement, then the "Quiet- 
Operation" must be enforced by the user when running the Builder. 
R-AlIClient: The Builder should provide functionality to create installation set(s) 
for each of the clients eStream 1 .0 is going to support. <Preferably there should be 
only one builder program that should recognize the OS it is running on and should 
create the appropriate installation set. Also if possible, we should be able to "diff ' 
installation sets for different OSs and if they are same, we should be able to create 



a single installation set for those OSs. The clients to be supported are W2K, 
WinNT4.0 and Win98>. 

• R-AppIdGeneration: It should be possible to change the appld of the eStream set 
when an ASP wants to "install" the eStream set in order to host it. Typically the 
builder will generate a default appld number for a new application which can be 
overridden by the ASP installer by using a Builder tool. 

• R-SuiteSupport: It should be possible to create a merged eStream set for a suite 
of applications. E.g. Office consisting of Word, Excel and Powerpoint. This could 
be done either by providing a tool for merging multiple eStream sets or by 
allowing the builder to serially monitor multiple installations in a session and then 
allowing the user to create a single package at the end of the session. 

• R-Testing: It should be possible to test the Builder using a stand-alone tester and 
not require the eStream client+server programs. 

• R-UpgradeSupport: The appInstallBlock should have support for indicating 
upgrades at the support site. E.g. When an eStream application is upgraded at the 
server (not as a separate app), the client will no longer be able to access/use it. We 
should provide some version of the appInstallBlock itself so that clients should 
detect that they will need to download the appInstallBlock again. 

• R-Manuallntervention: In the process of creating an eStream set it should be 
possible for the user to delete file entries and registry entries manually to "trim" 
the eStream set if she so desires assuming the user knows what she is doing. 

Issues 

• Profile Sequence Matrix is different for different machine configuration even if 
the user's usage pattern is the same. 

• Profile Sequence Matrix doesn't contain the right successor profile information as 
eStream cache is warmed up and pages from the cache is replaced. 

• Merging Module must take different machine configuration into account. Should 
this information be uploaded by the client as the same time it uploads the Profile 
Sequence Matrix to the Profile Server? 

• What is the difference between profiling based on the page sequencing seen by 
the eStream Cache Manager versus the page sequencing missed by the eStream 
Cache Manager? 
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Tricky Builder Issues 



Author: Sanjay Pujare 

This document enumerates all those tricky issues that may make the Builder's job 
difficult. Even though some solution may be proposed for some issues, not every issue 
would have a solution described in this document. The purpose of this document is 
mainly to keep track of Builder issues that may impose some limitations on the eStream 
technology. This way Omnishift marketing and deployment are aware of these 
limitations. 

1) The Builder cannot capture updates to existing files in an intelligent fashion (i.e. 
if the updates are based on a context or existing contents, it is very difficult to 
capture that). So the current Builder will just flag an error, if such an update 
occurs. 

Solution 

□ These updates are probably very rare, so we can defer it to the next 
release. 

□ For this release, we can try to solve this on a case by case basis e.g. we 
will try to solve this issue for INI files. 

□ Based on our understanding of general app installations, we might be able 
to make some generalizations that we can use in eStream e.g. Only certain 
files get updated; there is a definite pattern of updates. 

2) The current Builder drivers are based on the NT driver model, and hopefully we 
can implement the same functionality in the Win98 drivers, but this needs to be 
ensured. (This shouldn't be an issue, but. . .) 

3) We need to think some more about those cases when device drivers are installed 
by apps. Issues that can arise: 

□ This may not work correctly on the eStream client, just because the driver 
installation didn't take place properly. 

a The Builder would need to be able to figure out in an automated way if a 
client reboot is required or not. 

□ If the driver installation is h/w or s/w specific that can be difficult to tackle. 
Solution 

□ As we eStream more apps and gain more experience, we should be able to 
figure out solutions. 

4) There could be an ambiguity when the Installmon is trying to change absolute 
paths (or absolute values in general) to relative paths, e.g. A path like C:\WINNT 
can be changed to %SystemRoot% or %windir% since both of those environment 
variables are set to "C:\WINNT" on my system. 

Solution 

□ We can prioritize env vars and registry keys as described in the BuilderUI- 
LLD design document. 



□ We should encourage Builder operators to use as distinct values as 
possible for env variables and registry keys for the Builder machine. 
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Functionality 

The eStream Set is a data set associated with an application suitable for streaming over 
the network. The eStream Set is generated by the eStream Builder program. This pro- 
gram converts locally installable applications into the eStream Set. This document de- 
scribes the format of the eStream Set. 

Note: Fields greater than a single byte is stored in little-endian format. The eStream Set 
file size is limited to 2 A 64 bytes. The files in the CAF section are laid out in the same 
order as its corresponding entries in the SOFT table. 

Data type definitions 

The format of the eStream Set consists of 4 sections: header, Root Version Table (RVT), 
Size Offset File Table (SOFT), and Concatenation Application File (CAF) sections. 

1. Header section 

o MagicNumber [4 bytes]: Magic number identifying the file content with 

the eStream Set ^ 

o ESSVersion [4 bytes] : Version number of the eStream Set format. 

o AppID [16 bytes]: A unique application ID for this application. This 
field must match the AppBD located in the ApplnstallBIock. Guidgen is 
used to create this identifier. 

o Flags [4 bytes]: Flags pertaining to EStreamSet 

o Reserved [32 bytes]: Reserved spaces for future. 

o RVToffset [8 bytes] : Byte offset into the start of the RVT section. 

o RVTsize [8 bytes] : Byte size of the RVT section. 

o SOFToffset [8 bytes] : Byte offset into the start of the SOFT section. 

o SOFTsfee [8 bytes]: Byte size of the SOFT section. 

o CAFoffset [8 bytes]: Byte offset into the start of the CAF section. 

o CAFsize [8 bytes]: Byte size of the CAF section. 

o VendorNamelsAnsi [1 byte]: 0 if the vendor name is in Unicode format. 

1 if the vendor name is in ANSI format, 
o VendorNameLength [4 bytes]: Byte length of the vendor name, 
o VendorName [X bytes]: Name of the software vendor who created this 

application. I.e. "Microsoft" Null-terminated. 
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o AppBaseNamelsAnsi [1 byte] : 0 if the vendor name is in Unicode for- 
mat. 1 if the vendor name is in ANSI format. 

o AppBaseNameLength [4 bytes]: Byte length of the application base 
name. 

o AppBaseName (X bytes]: Base name of the application. I.e. "Word 

2000". Null-terminated, 
o MessagelsAnsi [1 byte]: 0 if the vendor name is in Unicode format. 1 if 

the vendor name is in ANSI format, 
o MessageLength [4 bytes]: Byte length of the message text, 
o Message [X bytes]: Message text. Null-terminated. 

2. Root Version Table (RVT) section 

The Root version entries are ordered in a decreasing value according to their file 
numbers. The Builder generates unique file numbers within each eStream Set in a 
monotonically increasing value. So larger root file number implies later versions 
of the same application. The latest root version is located at the top of the section 
to allow the eStream Server easy access to the data associated with the latest root 
version. 



o NumberEntries [4 bytes]: Number of patch versions contained in this 
eStream Set. The number indicates the number of entries in the Root Ver- 
sion Table (RVT). 



Root Version structure: (variable number of entries) 

o VersionNumber [4 bytes]: Version number of the root directory. 

o FileNumber [4 bytes] : File number of the root directory. 

o VersionNamelsAnsi [1 byte]: 0 if the vendor name is in Unicode format. 
1 if the vendor name is in ANSI format. 

o VersionNameLength (4 bytes]: Byte length of the version name 

o VersionName (X bytes]: Application version name. I.e. "SP 1". 

o Metadata [32 bytes]: See eStream FS Directory for format of the meta- 
data. 

3. Size Offset File Table (SOFT) section 



The SOFT table contains information to locate specific files in the CAF section. 
The entries are ordered according to the file number starting from 0 to Number- 
Files- 1 . The start of the SOFT table is aligned to 8 bytes boundary for faster ac- 
cess. 



SOFT entry structure: (variable number of entries) 

o Offset [8 bytes]: Byte offset into CAF of the start of this file. 
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o Size [8 bytes]: Byte size of this file. The file is located from address Off- 
set to Offset+Size. 

4. Concatenation Application File (GAF) section 

CAF is a concatenation of all file or directory data into a single data structure. 
Each piece of data can be a regular file, an AppInstallBlock, an eStream FS 
directory file, or an icon file. 

a. Regular Files 

o FileData [X bytes]: Content of a regular file 

b. AppInstallBlock (See AppInstallBlock document for detail format) 

A simplified description of the AppInstallBlock is listed here. For exact detail 
of the individual fields in the AppInstallBlock, please see AppInstallBlock 
Low-Level Design document. 

o Header section [X bytes]: Header for AppInstallBlock containing infor- 
mation to identify this AppInstallBlock. 

o Files section [X bytes]: Section containing file to be copied or spoofed. 

o Add Variable section [X bytes]: Section containing system variables to 
be added. 

o RemoveVariable section [X bytes]: Section containing system variables 
to be removed. 

o Prefetch section [X bytes]: Section containing pointers to files to be pre- 
fetched to the client. 

o Profile section [X bytes]: Section containing profile data, (not used in 
eStream 1 .0) 

o Comment section [X bytes]: Section containing comments about AppIn- 
stallBlock. 

o Code section [X bytes]: Section containing application-specific code 
needed to prepare local machine for streaming this application 

o LicenseAgreement section [X bytes]: Section containing licensing 
agreement message. 

c. EStream Directory 

An eStream Directory contains information about the subdirectories and files 
located within this directory. The information includes file number, names, 
and metadata associated with the files. 

o MagicNumber [4 bytes]: Magic number for eStream directory file, 
o ParentFilelD [16+4 bytes]: AppID+FileNumber of the parent directory. 
AppID is set to 0 if the directory is the root. 
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o SelfFilelD [16+4 bytes]: AppID+FileNumber of this directory. 

o NumFiles [4 bytes]: Number of files in the directory. 

o NumEntries [4 bytes]: Number of entries in the directory. Some entries 
are used for storing long file names and some are unused due to deleted 
files. So the NumEntries must be equal or less than NumFiles. 

Fixed length entry for each file in the directory consists of 2 formats (short 
format for storing files with name that fit the 8.3 convention; and long format 
for storing long file names). Each entry is 84 bytes and the entry are aligned 
on every 4K page boundry. Thus, in the first 4K page of the directory, the 
padding consists of 12 unused bytes (52 bytes for header + 48 entries * 84 
bytes per entry + 12 unused bytes = 4096 bytes). In all subsequent pages, the 
padding is 64 bytes (48 entries * 84 bytes per entry + 64 unused bytes = 4096 
bytes): 

Short Filename entry: 

o Format |1 byte]: Format of this entry, should be V for short format, T 

for long filename format, or possibly V, for unused, 
o ShortLen [1 byte]: Length of short file name, 
o LongLen [1 byte]: Length of long file name, 
o UNUSED [I byte]: Padding 

o NameHash (4 bytes): Hash value of the short file name. Algorithm TBD. 

o ShortName [24 bytes]: 8.3 short file name in Unicode 

o FilelD [16+4 bytes]: AppID+FileNumber of each file in this directory. 

o Metadata [32 bytes]: The metadata consists of file byte size (8 bytes), 
file creation time (8 bytes), file modified time (8 bytes), attribute flags 
(4 bytes), eStream flags (4 bytes). The bits of the attribute flags have 
the following meaning: 

■ Bit 0: Read-only - Set if file is read-only 

■ Bit 1: Hidden - Set if file is hidden from user 

■ Bit 2: Directory - Set if the file is an eStream Directory 

■ Bit 3: Archive - Set if the file is an archive 

■ Bit 4: Normal - Set if the file is normal 

■ Bit 5: System - Set if the file is a system file 

■ Bit 6: Temporary - Set if the file is temporary 
The bits of the eStream flags have the following meaning: 

■ Bit 0: ForceUpgrade - Used only on root file. Set if client is 
forced to upgrade to this particular version if the current root ver- 
sion on the client is older. 

■ Bit 1 : RequireAccessToken - Set if file require access token be- 
fore client can read it. 

» Bit 2: Read-only - Set if the file is read-only 

Long Filename entry: 

o Format [1 byte]: Format of this entry, should be T for long filename 
format. 



Omnishift Technologies, Inc. 



4 



Company Confidential 



eStream Set Format Low Level Design 

o Index [1 byte]: Number of this entry out of those used for this file's long 
name. 

o UNUSED [2 bytej: Padding 

o NameHash [4 bytes]: Hash value of the long file name. Algorithm TBD. 
o LongName [76 bytes]: Long filename in Unicode format. 

d. Icon files 

o IconFileData [X bytes]: Content of an icon file. 
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Open Issues 



o Where is the metadata associated with the Root directory located? Cur- 
rently, root metadata is located in the root version table. All other files 
and directory metadata can be found in their parent directory. 
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eStream BuilderUI Low Level Design 

Sanjay M Pujare 
<Date> 

Functionality 

The BuilderUI is the user interface part of the Builder. The operator uses this interface to 
use various functions provided by the Builder. Note that this UI may or may not be a 
graphical user interface. This low-level design is based on the assumption that a graphical 
user interface is not necessary. 

Data type definitions 
Interface definitions 
Component Design 

When the Builder UI is invoked with command line arguments which indicate that this 
was invoked by the Runonce mechanism of Windows, the control is transferred to the 
InstallMon: : startCaptureAf terReboot() function with the command line 
arguments passed as arguments to the function. When the Builder is invoked normally, it 
presents a menu which is managed by the function MainMenu. 

MainMenu 

This function manages the following menu hierarchy. Each 
menu option (leaf node) is followed by a function name in 
parentheses that is called to handle the option. 

1) eSt ream Set Menu 

1) New eStream Set (NewEStreamSet ) 

2) Open eStream Set (OpenEStreamSet) 

3) Save New/Upgraded eStream Set (EstreamSetCreation) 

2) Monitoring Menu 

1) Start Monitor (StartMonitor) 

2) Stop Monitor (StopMonitor) 

3) Check Status (CheckStatus) 

4) Inform Machine Reboot (Inf ormMachineReboot ) 

5) Get and Resolve Registry Set (GetRegistrySet ) 

6) Get and Resolve Files Set (GetFilesSet ) 

3) Profiling Menu 

1) Set the location of app executable (GetAppPath) 

2) Gather Initial Profile (Gatherlnit ialProf ile) 

4) eStream Set Creation Menu 

1) Set custom DLL (GetCustomDLL) 

2) Set User Comment (GetUserComment ) 

3) Set environment variables (GetEnvVars) 
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4) Set Reboot flag (GetRebootFlag) . 

5) Set License Agreement (GetLicenseAgreement ) . 

NewEStreamSet 

{ 

If there is an existing eStream set that hasn't been 
saved, warn the user. 

Get the following values from the user: 

• Name of the app setup program in gAppSetup 

• Dest directory where app will be installed in gDest- 
Dir (provide a default value) . 

• Dest location to store the new eStream set in 
gDestEstreamPath (provide a default value) . 

} 

OpenEStreamSet 

{ 

If there is an existing eStream set that hasn't been 
saved, warn the user. 

Get the following values from the user: 

• Location of the existing eStream set in gSrcEstreaim- 
Path (provide a default value) . 

• Whether the user wants to create an upgrade from this, 
or just wants to change the existing eStream set (gUp- 
grade) 

• If this is an upgrade (gUpgrade is true) , get all the 
values obtained by NewEstreamSet (i.e. gAppSetup, 
gDestDir, gDestEstreamPath) . 

Read the eStream set pointed to by gSrcEstreamPath; 
Load the existing file tables in gSrcCopiedFiles , 
gSrcSpoofedFiles and gSrcEFSFiles arrays; 



EstreamSetCreation 

{ 

If there is no working eStream set, give error and re- 
turn. 

if (gUpgrade) { 

Call UpgradeEStreamSet () with appropriate arguments; 

} 

else { 

Call CreateEStreamSet () with appropriate arguments; 

} 

if (gDestEstreamPath is not set) { 

assert (this is an update of an existing eStream set 
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and not an upgrade or a new eStream set creation) ; 
gDestEstreamPath = gSrcEstreamPath; 

} 

Save the eStream Set from memory to file to 
gDestEstreamPath; 

} 

StartMonitor 

{ 

If there is no working eStream set, give error and re- 
turn . 

if (gUpgrade) { 

Combine the gSrcSpoof edFiles and gSrcEFSFiles into an 
array f ileTableArray as expected by startCapture 
below; 

} 

Call InstallMon: : startCapture (gAppSetup, gDestDxr, 
gUpgrade, f ileTableArray) ; 

} 

StopMonitor 

{ 

If monitoring wasn't started, give error and return. 
Call InstallMon: : stopCapture ( ) ; 

} 

CheckStatus 

{ 

Ensure that monitoring was started and not stopped 
Call InstallMon: :checkSetupStatus () ; 

} 

InformMachineReboot 

{ 

Ensure that we are in the middle of monitoring. 
Call InstallMon: :machineToBeRebooted ( ) ; 

} 

GetReqistrvSet 

{ 

Call InstallMon: :getRegistryList () ; 

Store the set in gNewRegistry data structure; 

} 

GetFilesSet 

{ 

Call InstallMon: rgetFilesList () ; 
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Store the set in gNewFiles data structure; 
Also we need to capture changes made to INI files; since 
this has not been taken care of in the app install block 
and other parts of the Builder+Client we will need to 
make changes in all those components which are affected. 

} 

GetAppPath 

{ 

Ensure that all the InstallMon related data was captured. 
Get the location of the app that needs to be run 
to gather profiling info; 
Store it in gProf ileAppPath; 

} 

GatherlnitialProfile 

// this is the function that is used to get the initial 
// profile data (i.e. set of pages prefetched when an 
// eStream app is started for the first time) 
// implementation of this is yet to be defined 

} 

GetCustomPLL 

Get the location of the custom DLL file, validate that it 
is a DLL and store the path in gCustomDLLPath; 

} 

GetUserComment 

Get the user comment (optionally by browsing a text file) 
and store it in gUserComment ; 

} 

GetEnvVars 

{ 

Call the InstallMon: rsetEnvVars () function; 

} 

GetRebootFlag 

{ 

Until we come up with an algorithm to determine if a re- 
boot is required for an eStream app, get this value from 
the user. The default is FALSE: we do not want to reboot 
the client PC when the user subscribes to this eStream 
app. 
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} 

GetLicenseAqreement 

Get license agreement from the user. This could be: 

□ either, a default OmniShift license agreement 

□ or, a default ASP agreement 

□ or, license agreement that the app displayed. 

Let the user decide and enter the proper one. Provide a 
default based on our policy. 



Interesting issues to deal with: 

Testing design 
Unit testing plans 

Testing of the UI itself is a comparatively trivial task. The testing will basically consist of 
traversing the whole menu hierarchy. Since the menu is similar to a typical File Open -> 
File Edit -> File Save kind of a user application, this can be tested using simple hooks. 

Stress testing plans 

Since the Builder will be used in house at least initially only simple stress testing should 
be necessary. Make sure that Builder doesn't crash in the middle of processing so that we 
don't lose important data. Performance is not considered to be important. 

Coverage testing plans 

Basically the following 3 paths will be exercised: 

1. Create a new eStream set 

2. Open an existing eStream set to modify some data in it 

3. Open an existing eStream set to create an upgrade for it 

Cross-component testing plans 

Will be tested as a component of the whole builder. 

Upgrading/Supportability/Deployment design 

Deployment: This will be used in-house, so no deployment considerations. 
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Open Issues 

□ We need to think about the problem of converting absolute file paths discovered 
in the monitoring process to paths relative to some application or system registry 
key. Although most cases may not present a problem, we may have some difficult 
cases, which may make this problem non-automatable i.e. we may need some user 
intervention. Consider the case: 

KEYONE = C:\FOO\BAR 

KEYTWO - C:\FOO 

KEYTHREE = BAR 

If we notice that a file was copied to C:\FOO\BAR it won't be possible to convert 
this absolute path to a unique relative path since there are 2 solutions possible: 
%KEYONE% or %KEYTWO%\%KEYTHREE%. 

The way to solve this is by tracking only a set of well-known environment vari- 
ables and registry keys. Also in this set we prioritize all of them. So in the above 
case, %KEYONE% will be preferred over %KEYONE%\%KEYTHREE% just 
because of the way they were prioritized. 
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Functionality 

The File System and Registry Filter Driver (FSRFD) is a part of the Builder module that 
monitors file system and registry updates initiated by the Builder process. This driver just 
intercepts such requests and records them and returns the recorded data to the Install 
Monitor (INSTALLMON described in another LLD) program when requested by the lat- 
ter. All the intelligence, such as any decision-making logic, resides in the INSTALL- 
MON. 

For registry updates such as add or modify, the FSRFD needs to record the value added 
or modified. For registry deletes only the value name needs to be recorded. For file up- 
dates, there is no need to record the file contents added or modified, since the Builder 
would be interested only in the final contents of a file. 

Note: 

1. This does not cover those rare cases where an existing file is updated by an 
application install, and the eStream client would need to make the same up- 
dates. This kind of functionality is difficult to implement and will not be con- 
sidered for 1.0. 

2. The FSRFD will be used to monitor only one install at a time to simplify the 
design of the FSRFD. That means you cannot invoke multiple instances of 
the Builder at a time to monitor multiple installations. All Builder invoca- 
tions on a machine have to be strictly sequential. 

3. This design is based on the driver model for WinNT and Win2K. The Win98 
driver is not covered here (yet). 

Data type definitions 

The following struct is used to communicate information related to activating the 
FSRFD. Specifically, the process-id of the INSTALLMON and the 2 drives whose ac- 
cesses need to be monitored are passed. 

struct MonitorActivate_t { 

ULONG processld; // PID of INSTALLMON 
UCHAR sysDrive; // System drive letter 
UCHAR destDrive; // Dest drive letter 

}; 
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The following struct is used to return monitored data back to the INSTALLMON. Note 
that this is a variable size struct where the last field keyName is an array of one wide- 
char, but in reality is an array of length whose value is the sum of 3 length fields in the 
struct (nameLength, valueNameLength and dataLength). 

struct IMON_ENTRY { 

UCHAR regOrFile; // y R' for registry, 1 F' for files 

// and X E' for end of data 
UCHAR updateType; // "A' for add, *D' for delete, 

// *U' for update 
UCHAR valueType; // for registry only: value type 
// REG_SZ, REGJDWORD, REG__B I NARY , 

// REG_DWORD_LITTLE__ENDIAN, REG_DWORD_BIG_ENDIAN, 
// REG_EXPANDJ3Z, REG_LINK, REG_MULTI_SZ, REG_NONE, 
/ / REGJ2WORD , REG_QWORD_LITTLE_ENDI AN , 
// REG_RESOURCE_LIST 

ULONG narneLength; // length of name (file or 

// registry key) in wchars 
ULONG valueNameLength; // length of value name (if 

//it exists) in wchars 
ULONG dataLength; // length of data in bytes 

WCHAR keyName [1] ; // keyName followed by 

// valueName followed by 
// data: note none of these are 
// null terminated & are wide 
/ / chars 

}; 

Note about the updateType field: 4 A* is used for file creation and 'LP is used for any up- 
dates to the file. So if a 'IT is seen without an 'A' for a file that means the file was modi- 
fied but not created in this session. 

The following struct is used as the device extension in the FSRFD devices. Note that this 
extension is used for all device objects: the device that is created in the DriverEntiy func- 
tion to represent an "INSTALLMON" device for the INSTALLMON to access our driver 
as well as the devices that are created to create a filter layer above existing drives. 

enum DE V I CE_T Y PE { 

INS TALLMON I NTER FACE , 
STANDARD 

}; 
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struct DeviceExtension_t { 

PDEVICE_OBJECT deviceObj ect ; // device 

// for lower layer 
DEVICB_TYPE type; // see design 
KMUTEX pDeviceMutex; 
ProcessIdList 

// pointer to list of process-ids we are 

// interested in monitoring 

EntryList 

// This is the list that stores all the 
// info captured by the FSRFD until each 
// entry is queried by INSTALLMON 

}; 

There is an array for Fastlo that stores all the Fastlo function pointers which is required in 
a file system filter driver such as this. Note that we need to provide entry points for all (or 
most?) of the Fastlo routines in the dispatch table, since we need to pass the request down 
to the lower layer driver even if we are not interested in intercepting the request. For only 
some of the requests (e.g. all the FASTIO * WRITE* requests), we would be recording 
the file access and creating an entry to be returned to INSTALLMON. The Component 
Design section describes in more detail the FastIO routines that are implemented by this 
driver. 



Interface definitions 
INSTALLMON interfaces 



Since the FSRFD is a driver, it cannot provide directly callable APIs. Instead the IN- 
STALLMON communicates with the FSRFD using the DeviceloControl Win32 API. It 
uses Ioctl codes MON ACTIVATE, MON DEACTIVATE and MON GET ENTRY. 
The DeviceloControl API looks as follows: 



BOOL DeviceloControl ( 
HANDLE hDevice, 
DWORD dwIoControlCode, 
LPVOID IpInBuffer, 
DWORD nlnBuf f erSize, 
LPVOID lpOutBuffer, 
DWORD nOutBuf f erSize, 
LPDWORD lpBytesRe turned, 
LPOVERLAPPED IpOver lapped 



// handle to device 
// operation control code 
// input data buffer 
// size of input data buffer 
// output data buffer 
// size of output data buffer 
// byte count 
// overlapped information 



The semantics of each of the MONJ* codes is defined below. Note that this is described 
from the caller's (i.e. INSTALLMON) point of view. 
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Ioctl 1 - MONACTIVATE 

Input : 
hDevice : 

is the handle to the FSRFD device created. 

dwIoControlCode : 

The value MON_ACT I VATE 

lpInBuf fer : 

Address of MonitorActivate__t struct. This 
contains the process id of INSTALLMON. 

nlnBuf f erSize : 

Sizeof (MonitorActivate_t ) 

Output : 

lpOutBuf f er : 

Should be a ptr to a ULONG (at least) . 

nOutBuf f erSize : 

Size of above buffer 

IpBytesReturned : 

Should be a ptr to a DWORD where byte count 
of data returned in lpOutBuffer is returned. 

Comments : 

MON_ACTIVATE is sent to the FSRFD when the IN- 
STALLMON wants to start monitoring an installa- 
tion. MON_ACT I VATE can be sent only when the 
FSRFD is not already active - either after the 
driver is loaded the first time or after the 
last MON_DEACTIVATE request. 

Errors : 

□ STATUS_ALREADY_ACTIVE : FSRFD was already 
activated. The processld of the old acti- 
vation is returned in the ULONG pointed by 
lpOutBuffer. 

□ STATUS_INVALID_ARG: One of the arguments 
passed is not valid (either invalid Moni- 
torActivate_t ptr or processld) . 
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□ STATUS_INVALID_DRIVE: Either the sysDrive 
or the destDrive (or both) is invalid. 



Ioctl 2 - MON_DEACTIVATE 

Input : 

hDevice : 

is the handle to the FSRFD device created. 

dwIoControlCode : 

The value MON_DEACT I VATE 

IpInBuffer: 

NULL, or pointer to a ULONG where the 
ULONG has a non-zero value indicating a 
"forced" deactivation. A "forced" deacti- 
vation is done when the FSRFD still has 
entries that are not going- to be re- 
trieved. 

nlnBuf f erSize : 

0 or size of ULONG (depending on the 
above) . 

Output : 

No need for OUT arguments. 

Comments : 

MONJDEACTIVATE is sent to the FSRFD when the 
INSTALLMON wants to stop monitoring an instal- 
lation. MONJDEACTIVATE can be sent only when 
the FSRFD is already active i.e. after the last 
MON_ACTIVATE request. 

Errors : 

□ STATUS_NOT_ACTIVE : FSRFD was already deac- 
tivated. No special action is needed to be 
taken for this error condition. The caller 
can simply send a new MON_ACTIVATE . 

□ STATUS_PENDING_DATA: The FSRFD has some 
entries that were not read using 
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MONJ3ET_ENTRY . This error is only returned 
in case of non-forced deactivation. 

Ioctl 3 - MONGETENTRY 

Input : 
hDevice: 

is the handle to the FSRFD device created. 

dwIoControlCode : 

The value MON_GET_ENTRY 

IpInBuf f er : 
NULL. 

nlnBuf f erSize : 
0 

Output : 

lpOut Buffer: 

Should be a ptr to a IMON_ENTRY struct. 

nOutBuf f erSize : 

Size of above buffer 

lpBytesReturned : 

Should be a ptr to a DWORD where byte count 
of data returned in lpOutBuffer is returned. 

Comments : 

MON_GET_ENTRY is sent to get the next "entry" 
from the FSRFD. An "entry" is a record of a 
registry or file update intercepted by the 
FSRFD. The details are returned in the 
IMON_ENTRY struct passed in the lpOutBuffer ar- 
gument. Note that the FSRFD uses an event ob- 
ject (created using the IoCreateNotif ication- 
Event API) to signal the INSTALLMON that an 
"entry" is available to be read. INSTALLMON 
waits on this event object before retrieving 
the entry using M0NJ3ET_ENTRY . 

Errors : 
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□ STATUS_NOT_ACTIVE: FSRFD was not acti- 
vated. 

□ STATUS_INVALID_ARG: One of the pointer ar- 
guments passed is not valid. 

□ STATUS_INSUFF_BUFFER : The buffer size in- 
dicated by nOutBuf ferSize is insufficient 
for the current "entry" data. 

Ioctl4 - MON_GET_ERROR 

We need this to indicate occurrence of an error when- 
ever this occurs in any of the dispatch functions. We 
can either implement this control code or just return 
an error code for any MON_GET_ENTRY call that reflects 
that an error occurred. 

Event Object interface 

As mentioned above, an event object (lets call it JMON event object) will be used to sig- 
nal the INSTALLMON that a new entry is available. This event object will be created in 
the FSRFD in the processing of MONACTIV ATE ioctl, as: 

ext->pEvent = IoCreateNotif icationEvent ( 
L" \\BaseNamedObj ects\\lNS TALLMONE VENT " , 
ext->eventHandle); 

Whenever the FSRFD has a new entry, it signals using the above event object as follows: 

KeSetEvent (ext->pEvent , 0, FALSE) ; 

Whenever INSTALLMON gets the next entry using MONGETJENTRY ioctl, the 
FSRFD resets the event (if the list of entries is going to be empty after this entry) as fol- 
lows: 

KeClearEvent (ext ->pEvent) ; 
Whenever a MONJDEACTF/ATE is processed, the FSRFD will close the event as: 
ZwClose (ext->eventHandle) ; 

Kernel or Low Level Driver Interfaces 

Every driver needs a DriverEntry routine that is called when the driver is first loaded. 
This routine for FSRFD is described in the Component Design section. 
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The FSRFD inserts "hook routines" that intercept relevant registry and file system calls. 
The Component Design section describes which hook routines are inserted and what they 
do. 



Component Design 

Global variables 
plmonDevice 

This global variable points to the device object created in the DriverEntry function for the 
"\\Device\\installmon" device. 

DriverEntry 

NTSTATUS DriverEntry (IN DriverObject , IN RegistryPath) 

{ 

loCreateDevice for "\\Device\\installmon" ; 
plmonDevice = pDeviceObj ect returned above; 
ext = plmonDevice- >DeviceExtension ; 
ext->type = INSTALLMONINTERFACE ; 
IoCreateSymbolicLink with 

"\\DosDevices\\installmon" ; 
for all IRP_MJ_* values upto 

I RP_M J_MAX I MUM_FUNCT I ON { 
DriverOb j ect - >Ma j orFunction [ IRP_MJ_* ] = 
ImonDispatch 

} 

Setup the unload driver function 
DriverObject->FastIoDispatch = address of 
our fast io dispatch table ; 
Note that we are interested only in 
FAST_IO_*WRITE* routines for getting our 
entries , however we need to implement all 
of them to call lower layered drivers. All 
our Fastlo funcs are called ImonFastlo*; 
Create the necessary mutexes; 
Use PsSetCreateProcessNotifyRoutine to set a 
process create callback routine 
ImonProcessCallback; 



ImonProcessCallback 

{ 

// similar to ImonProcessCallback 

// This function is called every time a 
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// process on the system is created or 

// deleted. We need to figure out (by 

// checking the parent id in our list) 

// if we need to add or remove this process 

/ / from our list 

lock the mutex for ProcessIdList 
if not activated just return; 
if (this is process create) { 

Look up the parent process id in the 

ProcessIdList 

If present, add this process id to the 
ProcessIdList 

} 

else { 

if this process id is present then 
remove the process id 

} 

release the mutex 

} 

ImonDispatch 

{ 

// similar to FilemonDispatch 

// Instead of registering a different 

// function for each IRP_MJ_* this function 

// is called for all of them and this one 

// dispatches the right on based on the 

// control code 

This gets called for all IRP_MJ_*; 
if the device extension type tells us 
I NS TALLMON I NTERFACE 
call ImonDeviceFunc 

else 

call ImonHookFunc 

} 

ImonDeviceFunc 

{ 

// similar to FilemonDeviceRoutine 
// This function is called whenever an Ioctl 
// comes from the Installmon process that is 
// meant to be a command for this FSRFD. 
This is a request from the INSTALLMON using 
the INSTALLMON device. We would mainly be 
processing IRP_MJJ3EVICE_C0NTR0L, although 
ImonFast IoDeviceControl should have been 
called. So if we come here just call 
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ImonFastloDeviceControl 
Call IoCompleteRequest? 

} 

ImonHookFunc 

{ 

// similar to FilemonHookRoutine 
// This is the hook function that is called 
// for all the I/O requests that is made by 
// the I/O manager that need to go through 
// this driver (i.e. for those requests 
// that we are filtering) . 
We are interested in recording: 
IRP_MJ_CREATE where the Irp-> 

Parameters . Create . Options indicates 
a new file create (as opposed to an 
existing file open) 
IRP_MJ_WRITE 
Note that we have to get the current 
process-id using PsGetCurrentProcessId and 
look it up in the ProcessIdList (note: you 
have to exclude the first process-id since 
that belongs to the Installmon and not the 
setup program) and only if that search is 
successful, record the entry 
Note that we need to get the filename from 

the FileObject using code similar to 

FilemonGetFullpath 
Also we should be recording the entry when 
the request is successfully completed. So do 
this in the completion routine in a manner 
similar to filemon. 
To record: 

create a relevant IMON_ENTRY record; 

Call ImonAddEntry with this record; 
Pass all Irps to lower layer driver using 

IoCallDriver and getting the lower layer 

device object from this device's 

ext - >deviceOb j ect 

} 

ImonFastloDeviceControl 

{ 

// similar to both 

// FilemonFastloDeviceControl and 

// ImonDispatchloctl 

// This function gets called for all the 
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// IOCTLs. If it is from Installmon, we need 
//to process the MON_* commands or else 
// just pass on the command to the lower 
// layer driver. 

Get the current device's extension 
if type indicates INSTALLMONINTERFACE { 
switch (IoControlCode) { 
case MON_ACTIVATE : 
call ImonActivate; 
break ; 
case MON_DEACTIVATE : 
Cal 1 ImonDeac t i vate ; 
break ; 
case MON_GETENTRY : 
Call I monGe t En t ry ; 
break ; 

} 

} 

else { 

pass it down using deviceObject and the 
fastio hook for Ioctl 

} 

} 

ImonAddEntry 

{ 

// similar to ImonEnqueueRegEntry and 
// createRegEntry etc. 

// This function adds a IMON_ENTRY node 

// to our list: this list is eventually 

// returned to InstallMon 

create an entry rec from nonPagedPool 

Grab a mutex to modify EntryList 

Add the entry rec to EntryList 

release the mutex 

KeSetEvent for IMON event object 



ImonActivate 

{ 

// similar to ImonActivate and various 

// Filemon funcs called by 

/ / FilemonFastloDeviceControl 

// This is called by our own func that 

// handles IOCTLs when a MON_ACTIVATE is 

// sent by InstallMon 

Look at the ProcessIdList to ensure that we 
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are not already active (1 st process id) . 
If we are, return with an error with that 
process id 
Grab a mutex 

Add a node to ProcessIdList with the 

processld; 

Release the mutex; 

Create the IMON event object; 

HookDrive (sysDrive) ; 

HookDrive (destDrive) ; 

HookRegistry () ; 



ImonDeactivate 

{ 

// Same as above except for MON_DEACT I VATE 
If we are already deactivated 

return with an error; 
If EntryList is not empty and this is not 

a forced deactivation then 

return with error; 
Clear and Delete the IMON event objects- 
Free the ProcessIdList; 
Free the EntryList; 
UnhookRegistry ( ) ; 

} 

ImonGetEntry 

{ 

// When the InstallMon sends a MON_GETJENTRY 

// this function is called. 

Grab the mutex to access EntryList 

get next entry and remove from the 

list; 

if no entry then { 

if (deactivated and first process in 
the processIdList is dead) { 
Make a new entry of "end of data" 
type; 

} 

else { 

there is some problem (should not 
happen) 

} 

} 

copy the entry into the OUT buffer 
release the mutex; 



Omnishift Technologies, Inc. 



12 



Company Confidential 



eStream <COMPONENT> Low Level Design 



} 

HookDrive, UnhookDrive 

{ 

// very similar to Filemon's HookDrive 
// Hence not described here 
// Similarly UnhookDrive 

// When a MON_ACTIVATE comes, we need to 
// make sure that all the requests to the 
// system drive or dest drive are filtered 
// through us. e.g. If U C:" is the system 
// drive, HookDrive will register this 
// as the filter driver for all 10 for W C:" 
// Similary for UnhookDrive. 

} 

HookRegistry. UnhookReqistrv 

{ 

//similar to Installmon's (Un) HookRegistry 
//We need to make sure that all Registry 
// functions are replaced, with our funcs 
// so that these funcs get called whenever 
// a process is trying to access the 
// registry. Our hook funcs will in turn 
// call the real funcs after queueing an 
// entry 



ImonFastlo* routines 

{ 

// These are very similar to FilemonFast Io 
// routines except that we need to intercept 
// only FAST_IO_WRITE, 
// FAST_IO_MDL_WRITE_COMPLETE, 
// FAST_IO_WRITE_COMPRESSED, 
II FAST_IO_WRITE_C0MPLETE_COMPRESSED 
Call the lower layer driver; 
if the request was successfully completed { 
Record the details into our entry rec 
and enqueue it similar to how ImonHookFunc 
does it; 

} 

} 

Interesting issues to deal with: 
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□ Make sure that we use non-paged memory as required, e.g. all the nodes of the 
processIdList and entryList will be from non-paged pool. According to Bob, we 
do not need to always allocate non-paged memory - for Registry related nodes 
(i.e. when it is 'R' type IMONJENTRY node) we can allocate paged memory. 
However this needs to be checked - since there will be pointers between these 
nodes, we need to make sure that this will work properly. 

□ We have to make sure that a 2-phase install works with this: some setups ask you 
to reboot the machine and after the reboot the setup continues. For the FSRFD we 
need to make sure that after the reboot the FSRFD is already loaded before the 2 nd 
phase of the install starts. In the FSRFD we may need to make sure that when the 
"Runonce" registry key is updated (the installer is trying to do a 2-phase install 
and setting the 2 nd phase exe as the value of "Runonce") we capure that info and 
accordingly co-ordinate with the Installmon to do the right thing. To ensure that 
this driver is started at the right time on start-up (since the Builder machine is go- 
ing to be an in-house dedicated machine, it is okay), we need to add to the System 
registry (under 

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControl Set \ Services) 
appropriate values for Start, Group and Tag (and possibly others) value 
names. Actually this is going to be implemented in InstallMon as a user initiated 
event in which case the user informs the Builder UI (which in turn informs the In- 
stallMon) that a reboot is imminent. In that case the InstallMon can try to find all 
possible ways in which the setup.exe is achieving this: 

o The "RunOnce" key or one of its other incarnations (e.g. RunOnceService, 

RunOnceEx etc) has been modified. We need to figure out exactly which 

one of the "*RunOnce*" can be modified, 
o The setup.exe has actually added itself (or another exe) to the startup 

folder. If that is the case, we can do the same trick here: replace that entry 

with an entry pointing to the BuilderUI with the original setup.exe value 

as an argument to the BuilderUI. 

□ An issue that hasn't been resolved is any user interaction (and user input) that has 
taken place during installation that is being monitored. For example, an installa- 
tion may ask for a port number that it may store in a registry key. The solution 
suggested is as follows: This has to be a manual process. The Builder user should 
record all the manual interaction and manual data input that has taken place. He 
should recreate the same interaction in the custom DLL that the appInstallBlock 
provides for that app. This custom DLL at eStream app subscribe time can do the 
same thing that was observed during original application install. Alternatively we 
can request the ISV's co-operation in doing this. (May be this bullet should be 
transferred to a different doc such as the InstallMon LLD.). 

□ There is another unresolved issue about handling h/w or s/w dependent things that 
the installer does: we will have to handle this case by case basis and any knowl- 
edge we gain as a result of this, we should consolidate in the Builder components, 
e.g. we may notice that some installations may depend on IE4 or IE5 being there. 
Of course, one of the pre-requisites of the Builder is that it will be run on a pris- 
tine machines, so that we capture a maximal installation when it is taking place. 
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Testing design 
Unit testing plans 

This will be unit tested using the INSTALLMON program (or its early prototype). The 
INSTALLMON program sends all the required Ioctls like MONACTIVATE, 
MONDEACTF/ATE and MON GETENTRY. The INSTALLMON output will be used 
to check the correctness of the FSRFD. This means the INSTALLMON itself should be 
assumed to be correct. 

In addition to the above, we actually need to write one or more test programs that exer- 
cise the FSRFD. These test programs will be run as if they were App Installers i.e. as 
child processes of the INSTALLMON. The test programs should be written to exercise: 

□ All file systems (FAT, FAT32, NTFS, HPFS, compressed drives and others), 
since the Fastlo routines need to be tested. 

□ For each of the file systems: 

o File create (with and without the old file being there) 
o File update (existing file appended as well as modified in the middle) 
o • File touch (e.g. get the version of a DLL file) - this is not yet covered by 
this design 

□ Registry updates : 

o create a key 

o delete a subkey 

o delete a named value from a key 

o RegReplaceKey (we need to investigate if this is broken down into smaller 
reg calls). 

o RegRestoreKey (same comment as above applies) 

o RegSetKeySecurity (we failed to address this in the design) 

o RegSetValueEx 

o RegLoadKey and RegUnLoadKey 

Stress testing plans 

The FSRFD will be stress-tested using the above testing strategy by varying the rate at 
which files/registry are updated and under a variety of conditions (memory/disk, other 
processes). 

Coverage testing plans 

The unit testing above also covers Coverage testing. 
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Cross-component testing plans 

This will be tested with the Installmon program which is enough for cross-component 
testing. 

Upgrading/Supportability/Deployment design 

Deployment: This will be used in-house, so no deployment considerations. 

Open Issues 

These issues are not necessarily FSRPD related, but are listed here just to remind our- 
selves. 

□ Do we need to worry about environment variables? It looks like most installations 
(and their apps) won't be looking at environment variables. 

□ What about the .Ink files (shortcuts). It looks like the client guys will have to 
change all the .Ink files based on the actual client's settings. This issue has been 
addressed either in the InstallMon or Packager. Pis see those documents. (The 
IShellLink interface has been suggested). 

□ Also what about when device drivers are installed? There is no impact on the 
builder but the client will need to reboot and hopefully the existing applnstall- 
Block and the custom initialization dll should be able to take care of it. 
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eStream Instailmon Low Level Design 

Sanjay M Pujare 
<Date> 

Functionality 

The Instailmon is a part of the Builder module that talks to the FSRFD to monitor file 
system and registry updates initiated by the Builder process. The FSRFD driver just in- 
tercepts such requests and records them and returns the recorded data to Instailmon when 
requested by the latter. All the intelligence, such as any decision-making logic, resides in 
the Instailmon. 

Note: 

1 . This does not cover those rare cases where an existing file is updated by an 
application install, and the eStream client would need to make the same up- 
dates. This kind of functionality is difficult to implement and will not be con- 
sidered for 1.0. 

2. Somewhere in the docs (may be the user docs?) it should be mentioned that 
the Builder's (or rather Installmon's) job could be made easier by the user 
by following these guidelines: 

Make sure that values in all the registry keys are as distinct as possi- 
ble. We may need to create a special Win2K or WinNT installation 
where each key (or ValueName within a key) will be created with a 
distinct or unique value as far as possible. E.g. If the default Windows 
installation creates 2 keys FOO and BAR and stores the same value 
"C:\Windows\System32" in both of them, we wouldn't know which 
one of those keys is used when a file is copied to 
"C:\Windows\System32". To solve this problem, all the effort should 
be made to ensure that FOO and BAR have distinct values. 

- When the Builder operator is installing an app under the Instailmon, 
she should also make sure that the install script is given a distinct or 
unique value for each of the user inputs that may be used to set a reg- 
istry key or an environment variable. This is especially necessary 
when the inputs seem to be totally unrelated. For example, vendor 
name and installation directory name. If both are entered as "Micro- 
soft" then that could cause confusion to the Instailmon. 

3. There are 2 ways in which registry changes and file system changes can be 
captured: 

□ using a kernel mode driver such as the FSRFD, Or 

□ using a difPing mechanism for both the registry as well as the file system. 
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In this design, >ve use both the approaches, and record any differences due to 
these two approaches. We give the user a chance to manually edit the regis- 
try/file changes. 

Data type definitions 
Interface definitions 

All of the interfaces used to communicate with the FSRFD are described in the FSRFD 
LLD document. This LLD document will cover only interfaces with the other modules. 

Interfaces with the Builder UI 

The public methods of the InstallMon class described below are interfaces to the Builder 
UL Specifically these are: 

void InstallMon: : start Capture (PUNICODE_STRING setup_exe, 
PUNICODEJSTRING dest, bool upg, FileTable_t *fTbl) ; 

This function is called when the user chooses to start monitoring an install. This function 
is called after the user has entered all the necessary data such as the setup.exe path etc. 

void InstallMon: rstopCapture () 

This is called in rare cases when the user wants to terminate a running install in abnormal 
cases. 

bool InstallMon: : checkSetupStatus () 

This is used by the UI to check the status of the install: whether the file list is ready to be 
processed and the registry list is ready to be processed etc. 

void InstallMon: :getRegistryList ( ) 

This function is used to finally get the list of registry changes that occurred after the in- 
stall has taken place. This function allows the user to edit the list to manually 
add/delete/modify entries to be able to override. 

void InstallMon: :getFilesList () 

This function is used to finally get the list of file system changes that occurred after the 
install has taken place. This function allows the user to edit the list to manually 
add/delete/modify entries to be able to override. 

void InstallMon: : machineToBeRebooted () 

When the app install program is asking the user to reboot the machine (in the middle of 
the install) to continue installation, the user has to inform the Builder UI that a reboot is 
imminent. The InstallMon does the necessary bookkeeping to make sure that monitoring 
continues after the reboot. 

void InstallMon: : startCaptureAf terReboot (setup_exe / dest, 
upg) 
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When the installation continues after the reboot, the Builder UI is automatically invoked, 
and it calls this function to inform the InstallMon to continue monitoring. 

Access DB interface 

We will be using an Access DB (or alternately the MSI databases as suggested by 
Bhaven) to store intermediate results of the Installmon process. There will be 2 tables 
used: Registry and Files. 

Registry table 

Fullpath : string; // this is the full path of the key 

ValueName : string; // this is the value name: for 

// (default) use NULL 
UpdateType: char; // Add/update or delete, also 

// X R' for removed 
// combined (Fullpath, ValueName) should be unique i.e. 
// primary key 

Files table 

Fullpath : string; 

UpdateType : char; // add, update 

Kind: char; // x C'opied, ^S'poofed, % E' stream 
Fileld: Number; 

// Combined (Fullpath, UpdateType, Fileld) should be unique 



RegistryDiff table 

Fullpath : string; 
ValueName : string ; 
ValueType : char ; 

Value: something that can store all REG_* types 
status: char; // x O'riginal, 1 C (add/change) , X D' 
// combined (Fullpath, ValueName) should be unique 

FjjesDjff table 

Fullpath: string; 

Type: char; // file *F' or dir *D' etc 

Status: char; // 'O'riginal, . ..'C, *A' , *D' etc 

// (Fullpath 

Component Design 

struct FileTable_t { 

PUNICODE_STRING name; 

Char type; // spoofed, copied or eFS 
Int fileld; 

}; 



class InstallMon { 
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PUNICODEJSTRING setup, destDrive; 
bool interruptThread; 

EventObject signalMonitor , signalSetup; 

bool rebootReq = false; 

bool afterReboot = false; 

bool completed; 

int exitCode; 

bool upgrade; 

some Type envVars; 

// TODO: combine the above 2 into an enum 
int upgradeFileldBegin = -1; 
int currFileld; 

static threadSetup (InstallMon iMon) { 

Remove all the environment variables except 
the ones in iMon- >envVars; 
Start the Setup program; 
And wait for it to finish; 

iMon->exitCode = exit code from the process; 
when finished signal the signalSetup event; 

static threadMoni tor (InstallMon *iMon) { 

// this is the func that runs as a separate - 
// thread, until we are interrupted or we 
// notice that the setup process has exited, 
get the current 

process id, system drive (using 
GetSystemWindowsDirectory) , destDrive and send 
the MON_ACTIVATE message> 
Start threadSetup thread with setup_exe; 
processEnded = false; 
while (! interruptThread) { 
if (rebootReq) { 

only poll for Installmonevent; 
if (signal not set) { 

break from the while loop; 

} 

} 

else { 

WaitForMultipleObjects -> signalMonitor, 
Installmonevent and signalSetup; 

} 

switch (signal) { 
case Installmon event: 

get the MonitorEntry_t record; 

switch (regOrFile) { 

case *R' : 

switch (updateType) { 
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case X A' : 
case. X U' : 

add or update (KeyName, ValueName, 
*N') to 
registry tables- 
break; 
case *D' : 

add (KeyName, ValueName, X D') to 
registry table; 

} 

break; 
case *F' : 

switch (updateType) { 
case 'A' : 

// file creation 

add (fullpath, X A' , *?', 

iMon->currFileId++) to files 
tables- 
break; 
case % U' : 

add (fullpath, > U / ) to files 

tables- 
break; 

} 

breaks- 
case *E' : 

no more data to add; 
if ( iprocessEnded) { 

// something wrong! ! ! ! 

} 

break out of the while loop 
break; 

} 

breaks- 
case setupevent : 

processEnded = true; 
Send M0NJ3EACTIVATE to the FSRFD; 
breaks- 
case signalMonitor event: 
if (rebootReq) { 

kill the threadSetup thread 
clear the signalMonitor event; 

} 

else { 
// TBD? 

} 

break; 
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} // switch 

} // while 

if (processEnded) { 

// normal setup process completion 

// registry and files tables should have 

// all the captured data from the FSRFD 

iMon->dif f Capture () ; 

iMon->completed = true; 

} 

} // threadMonitor 

void commonCapture (setup_exe, dest) { 
setup = setup_exe; 
destDrive = dest; 

Start the threadFunc thread, and pass *this' to 
it; 

} 

void diff Capture () { 

// Bhaven suggested that we could instead do an export 

// from regedit and do a text diff for registry diffs. 

// similarly: It seems the regular Unix diff tool also 

// compares directories. I don't know if it is 

// recursive. If it is, we might be able to use it. 

// Further investigation is recommended for doing 

// files diff. 

Query the OTI \BUILDERSTART key and get the value 

in builderStartTime and delete the 

OT I \ BU I LDER S TART key; 

// process the registry 

// step 1: query RegistryDiff table 

for each row in RegistryDiff table { 

query the corresponding key+valueName in the 

Windows registry 

if (exists) { 

if (no change to value) { 

update the row status to *U' for unchanged 

} 

else { 

update the row status to % C (changed) 

} 

} 

else { 

update the row status to A D ' (deleted) 

} 

} 

// step 2: enumerate the whole Windows registry 
for each enumerated registry key+value { 
query the RegistryDiff table; 
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if (a row exists) { 
if (status is »U' - ) { 
delete the row 

} 

else { 

status should be 1 C and values should be 
different bet table and actual registry 
or else display fatal error (?) 

} 

} 

else { 

// no previous value 

add a row to RegistryDiff with (fullpath, 

valueName, ValueType, Value, 'A') to mark 
it as an added registry key 

} 

} 

// now repopulate the FilesDiff table 
// step 1: query FilesDiff table 
for each row in FilesDiff table { 

query the corresponding Fullpath in the actual 
file system; 
if (exists) { 

if (no change (i.e. timestamp before 
builderStartTime) ) { 

update the row status to X U' for unchanged 

} 

else { 

update the row status to % C (changed) 

} 

} 

else { 

update the row status to *D' (deleted) 

} 

} 

// step 2: traverse the whole file system 
for each file/dir in { systemDrive, destDrive} { 
query the FilesDiff table; 
if (a row exists) { 
if (status is % U' ) { 
delete the row 

} 

else { 

status should be X C and the file/dir 
timestamp should be after builderStartTime 
or else display fatal error (?) 

} 
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} 

else { 

// no previous value 

add a row to FilesDiff with (fullpath, 

% F' or *D' , X A') to mark it as an added 
f ile/dir ; 

} 

} 

} 

void recursiveFileGetter (curDir) { 

add a row to FilesDiff as (curDir, *D' , *0'); 
for (each element of curDir) { 
if (element is a dir) { 

recursiveFileGetter (element ) ; 

} 

else { // has to be a file(?) 

add a row to FilesDiff as (element, 'F', 
x O') ; 

} 

} 

} 

public : 

InstallMon(?) ; 
void setEnvVars ( ) { 

allow the user to set environment variables 
in an interactive way; 
Get the values in envVars; 

} 

void startCapture (PUNICODE_STRING setup_exe, 

PUNICODE_STRING dest, bool upg, FileTable_t *fTbl) { 

clear the Access DB registry, files, 
registryDiff and FilesDiff tables; 

if (upg) { 

populate the files table with data from the 

f Tbl array; 
upgradeFileldBegin = last file id + 1; 
currFileld = upgradeFileldBegin; 

} 

else { 

currFileld = 0 or 1 (depends on where to start) ; 

} 

Query the whole registry and 

for (each key and valueName pair) { 

add (key, valueName, valueType, value, y 0') to 
registryDiff ; 

} 

Get the current time and store it in some format 
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in a registry key OTI \BUI LDERSTART ; 

for (curDrive in { systemDrive, dest}) do 

{ 

Root = root of curDrive (e.g. "C:\\") ; 
recursiveFileGetter (Root ) ; 

} 

interruptThread = false; 
afterReboot = false; 

commonCapture (setup_exe, dest, upg) ; 

} 

void stopCapture () { 

// abnormal capture termination 
// TBD 

// also there might be normal cases where this 
// func is called when the setup process exit 
// is not detected for some reason (or doesn't 
// happen) ! May be we don't have to worry about 
// these things. 

} 

bool checkSetupStatus () { 
if (! completed) { 

display error and return false; 

. } 

if (exit Code is not okay) { 

display error and return false; 

} 

} 

void getRegistryList () { 

checkSetupStatus () and conditionally return; 
Using appropriate SQL, show (fullpath, ValueName) 
tuples that exist in Registry but not in 
RegistryDif f; 
Tell the user that these were captured by FSRFD 
(installmon) driver but not by the diff process ; 
If user chooses to remove any tuples from the 
shown list, mark these with status as X R' in the 
Registry table; 

Using appropriate SQL, show (fullpath, ValueName) 
tuples that exist in RegistryDif f but not in 
Registry; 

Tell the user that these were captured by diff 
but not by the FSRFD (installmon) driver; 
If user chooses to add any tuples from the 
shown list, add these tuples to the Registry 
table with status copied from RegistryDif f ; 
Remove all the rows from the Registry table 
where status is *R' ; 
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Now we have all the correct entries in Registry; 
Read each row of Registry and into an array to be 
passed to the caller (most probably the caller of this 
func would have passed an array in which we should 
pass these rows) ; 

} 

void getFilesList ( ) { 

checkSetupStatus () and conditionally return; 
Using appropriate SQL, show (fullpath) 
tuples that exist in Files but not in 
FilesDif f; 

Tell the user that these were captured by FSRFD 
(installmon) driver but not by the diff process; 
If user chooses to remove any tuples from the 
shown list, mark these with status as *R' in the 
Files table; 

Using appropriate SQL, show (fullpath) 

tuples that exist in FilesDif f but not in 
Files; 

Tell the user that these were captured by diff 
but not by the FSRFD (installmon) driver; 
If user chooses to add any tuples from the 
shown list, add these tuples to the Files 
table with status copied from FilesDif f; 
Remove all the rows from the Files table 
where status is 'R' ; 

Now we have all the correct entries in Files; 

Using appropriate SQL, select files (and not 
dirs) where the file has only X U' and not % A' : 
this means the setup modified an existing' file 
and we cannot handle this; so if this happens 
show a fatal error ; 

Read each row of Files and classify it into 'C, 
*S' or 'E' based on the following logic: 
If the file belongs to the app install directory 
(or app drive?) it is an *E'; 

If the file is smaller than a threshold then 
it is % C or else it is *S'; 

if (upg) { 

we need to create new file-ids for directories 

that contain new files or directories 

int prevStart = upgradeFileldBegin; 

int prevEnd = currFileld; 

while ( (prevEnd - 1) > prevStart) { 
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using appropriate SQL, select rows from the files 
table where fileld >= prevStart and kind is not 
% C; 

for each such row { 

fullpath = fullpath in the row; (e.g. "C:\A\B") 
dir = parent directory of fullpath (e.g. 
U C : \A" ) 

select in files a row where fullpath = dir and 
fileld >= upgradeFileldBegin; 
if (doesn't exist) { 

add in Files a row with (dir, *X' , *?', 
currFileId++) ; 

// Here V X' stands for u new file id for a 

// directory created because one of the 

// children has a new file id 

} 

} 

prevStart = prevEnd; 
prevEnd = currFileld; 

} 

} 

Also for each file, change the absolute path to 

an envVar relative or registry-value relative 

one: this is a non- trivial task. The way to do this is 

proposed below. However we need to refine this 

algorithm based on actual Builder runs on real apps : 

For a basic Win2K (or WinNT as the case may be) system 
create a list of registry keys (and env vars) whose 
values are normally used as destination paths for 
copying various kinds of files. To this list also add 
this app's dest dir as one of the values. Also add all 
the registry keys/values added as part of this app's 
install. Create an access table that looks like: 

Key: String; // registry or env or other name 

Type: char; // *R'egistry, *E'nv, *D'est dir etc. 

Source: char; // 'S'ystem, x A'pp, 'U'nknown? 

Value: string ; 

Now all of the *E' files should be relative to dest 
dir (i.e. X D' type above). Also any sub-directory 
strings should either be hard-coded or based on some 
registry key values where the registry key is of 
source 'A' (app) above. 
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All the X C and 'S' files should be relative to one of 
the registry keys of source X S' above and preferably 
type 'R' (since environment variables are not used un- 
der Windows?) . 

If there is a sacred file set (i.e. files that are 
so "sacred" for eStream apps, that they were installed 
on the client when the eStream client was installed) , 
we need to remove those files from this list; 

Set these values in the table and read the whole 
table in an array (most probably the caller of 
this func would have passed an array in which we 
should pass these rows) ; 

} 

void machineToBeRebooted () { 

//The setup is going to reboot the machine. 

//Note that there is a race condition possible here. 

// Suppose the setup.exe has displayed a dialog 

// asking the user to confirm a reboot. At this time 

// the setup.exe has still not written to "Runonce" . 

// Only after the user has confirmed (by pressing 

// Okay) will the program write to "runonce" and 

// auto reboot. In that case, this function will be 

// called at the wrong time. We may need to modify 

/ / the FSRFD to detect changes to "Runonce" . 

rebootReq = true; 

signal the signalMonitor event; 

wait for the threadMonitor thread to end; 

Query either our Access database or the actual 

registry to see if Runonce key is set/updated. 

if (not) { 

we cannot handle this reboot situation; 

give fatal error; 

// may be we can look at the 

// Start\Programs\Starup folder and do the 

// same thing we did with Runonce key 

} 

OldRunonce = Old value of Runonce key; 
Set the new value of Runonce key as our Builder 
name (or whichever EXE has the installmon code) 
followed by OldRunonce as the argument to that 
EXE and the destDrive as the next argument and 
upgrade as the next arg; 

Prompt the user to go ahead and say Okay to 
Setup's dialog warning that it is going to 
reboot ; 
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void startCaptureAf terReboot (setup__exe, dest, upg) { 
afterReboot = true; 

commonCapture (setup__exe, dest, upg) ; 

} 

}; 

Interesting issues to deal with: 

Testing design 
Unit testing plans 

The unit testing of installmon will be done in conjunction with the FSRFD unit testing. 
This has been described in the FSRFD-LLD document. In addition, we will be using the 
sysdiff tool to validate the results of the Installmon. 

Stress testing plans 

All of the 14 or so applications (that OTI is planning to convert to eStream) installs will 
be tested under the installmon. Any issues found will be used to fix and improve the in- 
stallmon functionality. 

Coverage testing plans 

The testing of the Builder/installmon on the 14 or so app installs should give us enough 
coverage. Considering that Builder/installmon will be used in-house for some time makes 
some of the testing issues less significant. 

Cross-component testing plans 

Will be tested as a component of the whole builder. 

Upgrading/Supportability/Deployment design 

Deployment: This will be used in-house, so no deployment considerations. 

Open Issues 

□ On one of the newgroups someone mentioned a key 
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\SharedDLLs] that 
we can look at for the shared DLLs that cannot be installed. Need to investigate. 

□ 2-phase install: some installs need a reboot after which they continue. The key to 
look at is Runonce (under the same path as SharedDLLs I think). 
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Functionality 

The eStream Application Builder Package Manager is responsible for packaging data 
gathered from the Installation Monitor, the Profile Manager, and the Upgrade Monitor 
into a set of data called the eStream Set. For the detail format of the eStream Set, see the 
separate document on eStream Set. 

The Package Manager must perform the following task: 

□ Create the appInstallBlock containing C-File and Registry data from the Install 
Monitor; Prefetch data from the Profile Manager; and Updated C-File and Up- 
dated Registry data from the Upgrade Monitor 

□ Create a custom installation DLL needed by a specific applications and add to the 
appInstallBlock 

□ Create directory files associated with each directory of the application director 
and add metadata to the directory 

□ Create directory files associated with each Windows directory containing both the 
Spoofed files and Z-files 

□ Create Concatenated Application File (CAF) which is just a juxtaposition of the 
application files, eStream directory files, and AppInstallBlock 

□ Create Size Offset File Table (SOFT) which is a mapping of fileNumber to offset 
of the start of the CAF file 

□ Create Root Version Table (RVT) which is a mapping from the version of root to 
the file number of the root directory file 

□ Archive the CAF, SOFT, and RVT into a single structure called eStream Set suit- 
able for uploading to the eStream Servers. 

Data type definitions 

The Package Manager doesn't have any internal data types. It must accept and under- 
stand data structures received from the Install Monitor and the Profile Manager. See In- 
stall Monitor and Profile Manager components for the description of the data structures. 

The Install Monitor is responsible for generating the following list of information: list of 
copied-files, list of spoof-files, list of files with file numbers, list of add registry entries, 
and list of delete registry entries. The list of copied-files contains the files copied into ' 
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non-application specific directories. The list of spoof-files consists of the files too large 
to be downloaded to the client in the AppInstallBlock. Those files are copied into some 
special directory on the Z drive for streaming. The list of files with file numbers consists 
of the files copied into the standard "Program Files" directory and the files that will be 
spoofed. The registry information is a list of registry key added or removed during the 
installation of the application. 

Struct FilelndexTable { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING FilePathName; 
ULONG FileNumber; 
} Entries [NumEntries] ; 

}; 

Struct FileCopied { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING FilePathName; 
} Entries [NumEntries] ; 

} 

Struct FileSpoofed { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING OldFilePathName ; 
PUNICODEJSTRING NewFilePathName ; 
} Entries [NumEntries] ; 

}; 

Struct Registrylnfo { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING KeyName; 

PUNICODEJSTRING ValueName; 

FVALUEJDATA ValueData; 
} Entries [NumEntries] ; 

>;' 

Struct Inilnfo { 
UINT NumFiles; 
Struct FileEntry { 

PUNICODE_STRING FilePathName; 
UINT NumSections; 
Struct SectionEntry { 

PUNICODE_STRING SectionName; 
UINT NumValues; 
Struct Entry { 

PUNI CODEJS TRING ValueName; 
PVALUE_DATA ValueData; 
} Entries [NumValues] ; 
} Entries [NumSections] ; 
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} Entries [NumFiles] ; 

}; 

The Profile Manager generates AccessCounts and the PrefetchBlocks data with the struc- 
tures shown below. 

Struct AccessCounts { 
UINT NumEntries; 
Struct Entry { 

PUNICODEJSTRING FilePathName; 
ULONG Frequency; 
} Entries [NumEntries] ; 

}; 

Struct PrefetchBlocks { 
UINT NumEntries; 
Struct Entry { 

PUNICODE_STRING FilePathName; 
ULONG BlockNumber; 
} Entries [NumEntries] ; 

}; 

The eStream Set has the following data structure (described in more detail in the separate 
eStream Set document): 

Struct eStreamSet { 

Struct eStreamSetHeader header; 
Struct eStreamSetRVT rvt; 
Struct eStreamSetSOFT soft; 
Struct eStreamSetCAF caf; 

}; 

Interface definitions 
Function 1 : CreateEStreamSet 

// Create the initial eStream Set from the data 
//retrieved from the Install Monitor and the 
// Profile Manager. 

// This function is called only by the Builder 

// UI after data is obtained from Install 

// Monitor and Profile Manager. 

int CreateEStreamSet ( 

IN PFILE_INDEX_TABLE FIT, 

IN PFILE_SPOOFED Spoof Files, 

IN PFILE_COPIED CopiedFiles, 

IN PREGI STRY_INFO AddRegistry, 

IN PREGI STRY_INFO RemoveRegistry , 

IN PINI__INFO Inilnfo, 

IN PACCESS_COUNTS AccessCounts, 

IN PPREFETCH BLOCKS PrefetchBlocks, 
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IN PVOID DllCode, 

IN PUNICODE_STRING Comment, 

OUT PESTREAMJSET EstreamSet) 

Input : 

FIT: File Index Tree contains the file 
number of the directories, spoofed 
files, and standard files 

CopiedFiles: pointer to a list of files 
To be copied to AppInstallBlock 

Spoof Files: pointer to a list of files 
To be spoofed on the client 

AddRegistry: pointer to a list of registry 

Data to add 
RemoveRegistry : pointer to a list of 

Registry data to remove 
Inilnfo: pointer to a list of ini changes 

AccessCounts : pointer to the list of 
Files with the access frequency 

Pref etchBlocks : pointer to the prefetch data 
To be inserted into the appInstallBlock 
Of the eStream Set 

DllCode: pointer to DLL Code 

Comment: pointer to comment string 

Output : 

EstreamSet: pointer to the eStream Set 

Return Value: 

Success or failure of the packaging process 

Comment s : 

The eStream Set will be large for most 
application. Intermediate data will be 
stored on the local hard-drive. 

Errors : 

OutOf Storage : failure to find enough storage 
For this eStream Set 
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FileNotFound: failure to find the files 
Specified by either ListCFiles or 
ListZFiles 

Function 2 : UpgradeEStreamSet 

// Upgrade the eStream Set to the latest 

// version. This function is only called by 

// the Upgrade Manager within the same process. 

int UpgradeEStreamSet ( 

INOUT PESTREAM_SET EstreamSet, 
IN P F I LE__ I NDEX_TAB LE UpgFIT, 
IN PFILE_SPOOFED UpgSpoof Files , 
IN PFILE_COPIED UpgCopiedFiles , 
IN PREGISTRY_INFO UpgAddRegistry , 
IN PREGISTRY_INFO UpgRemoveRegistry , 
IN PACCESS_COUNTS UpgAccessCounts , 
IN PPREFETCH_BLOCKS UpgPref etchBlocks , 
IN PVOID UpgDllCode, 
' IN PUNICODEJSTRING UpgComment) 

Input : 

UpgFIT: File Index Tree contains the file 
number of the directories, spoofed 
files, and standard files 

UpgCopiedFiles: pointer to a list of files 
To be copied to AppInstallBlock 

UpgSpoof Files : pointer to a list of files 
To be spoofed on the client 

UpgAddRegistry: pointer to a list of 
Registry data to add 

UpgRemoveRegistry: pointer to a list of 
Registry data to remove 

UpgAccessCounts: pointer to the list of 
Files with the access frequency 

UpgPref etchBlocks : pointer to the prefetch 
Data to be inserted into the 
AppInstallBlock Of the eStream Set 

UpgDllCode: pointer to DLL Code 
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UpgComment : pointer to comment string 
Output : 

EstreamSet: pointer to the eStream Set 

Return Value : 

Success or failure of the packaging process 

Comments : 

The eStream Set will be large for most 
application. Intermediate data will be 
stored on the local hard-drive. 

Errors : 

OutOf Storage : failure to find enough storage 
For this eStream Set 

FileNotFound : failure to find the files 
Specified by either ListCFiles or 
ListZFiles 

Function 3 : ModifyEStreamSet 

// Insert new data into different sections 
//of estream set. This function is overloaded 
// to handle different section data 

int ModifyEStreamSet ( 

INOUT PESTREAM_SET EstreamSet, 

IN PFILE_INDEX_TABLE FIT, 

IN PPREFETCH_BLOCKS Pref etchBlocks) 

int ModifyEStreamSet ( 

INOUT PESTREAMJSET EstreamSet, 
IN PREGISTRY_INFO AddRegistry, 
IN PREGI3TRY_INF0 RemoveRegistry) 

int ModifyEStreamSet ( 

INOUT PESTREAM_SET EstreamSet, 
IN PFILE_INDEX_TABLE FIT, 
IN PFILE_SPOOFED Spoof Files, 
IN PFILE_COPIED CopiedFiles) 

int ModifyEStreamSet ( 

INOUT PESTREAM_SET EstreamSet, 
IN PVOID DllCode) 

int ModifyEStreamSet ( 
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INOUT PESTREAM_SET EstreamSet, 
IN PUNICODE_STRING Comment) 

int ModifyEStreamSet ( 

INOUT PESTREAMJSET EstreamSet, 
IN PUNICODE_STRING LicenseText) 

Input : 

FIT: File Index Tree contains the file 
number of the directories, spoofed 
files, and standard files 

CopiedFiles: pointer to a list of files 
To be copied to AppInstallBlock 

Spoof Files: pointer to a list of files 
To be spoofed on the client 

AddRegistry: pointer to a list of registry 

Data to add 
RemoveRegistry : pointer to a list of 

Registry data to remove 
Inilnfo: pointer to a list of ini changes 

AccessCounts : pointer to the list of 
Files with the access frequency 

PrefetchBlocks : pointer to the prefetch data 
To be inserted into the appInstallBlock 
Of the eStream Set 

DllCode: pointer to DLL Code 

Comment : pointer to comment string 

Output : 

EstreamSet: pointer to the new eStream Set 

Return Value : 

Success or failure of the insertion process 

Comments : 

The eStream Set will be large for most 
application. Intermediate data will be 
stored on the local hard-drive. 

Errors : 
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OutOf Storage : failure to find enough storage 
For this eStream Set 

FileNotFound: failure to find the files 
Associated with the prefetch blocks 

Component design 

The pseudo-code for the function CreateEStreamSet is described below: 

{ 

Create AppInstallBlock (AIB) from the following input files: 
o SpoofFiles 
o CopiedFiles 
o AddRegistry 
o RemoveRegistry 
o Prefetch 
o Comment 
o DLLcode 

Assign AppInstallBlock with a unique fileNumber given by the IM; 
Record Root fileNumber in the first entry of Root fileNumber Table (RFT); 
Move AppInstallBlock under the Root directory by adding a new entry in the 

Directory structure; 
Create a Concatenation Application File (CAF) header; 
Create a Size Offset File Table (SOFT) header; 
For each (file in FIT) { 

If (file is a directory) { 

Create the directory with new list of fileNumber, filename, and 
Metadata; 

} Else { 

Find the file in the proper location on the HD; 

} 

Append the file or directory to the end of the CAF file; 

Append the fileNumber, offset into CAF, and size of file in SOFT; 

} 

Archive CAF, SOFT, and RFT into a single eStream Set; 
Return eStream Set; 

} 

The pseudo-code for the function UpgradeEStreamSet is mentioned below: 



Omnishift Technologies, Inc. 



8 



Company Confidential 



eStream Builder Package Manager Low Level Design 



{ 

Extract previous version PrevAppInstallBlock from eStream Set; 
Create new ApplnstallBlock with new FileNumber; 

Extract PrevSpoofFiles and PrevCopiedFiles from PrevAppInstallBlock; 
Divide the C-Files into SpoofFiles and CopiedFiles; 
Add PrevSpoofFiles to SpoofFiles; 
Add PrevCopiedFiles to CopiedFiles; 

Extract PrevAddRegistry and PrevRemoveRegistry data from 

PrevAppInstallBlock; 
Add any unique ((UpgAddRegistry plus PrevAddRegistry) minus 

UpgRemoveRegistry) in the new ApplnstallBlock AddRegistry section; 
Add any unique ((UpgRemoveRegistry plus PrevRemoveRegistry) minus 

UpgAddRegistry in the new ApplnstallBlock; 

Add UpgPrefetch data to new ApplnstallBlock; 
Add UpgDllCode data to new ApplnstallBlock; 
Add UpgComment data to new ApplnstallBlock; 

For each (directory in UpgFIT) { 

If (any child fileNumber has changed) { 

Create new directory with updated fileNumber; 

Append file to end of Concatination Application File (CAF); 

Append Size Offset File Table (SOFT) with new entry; 

} 

} 

Append new ApplnstallBlock to the end of CAF file; 

Prepend Root FileNumber Table (RFT) with new Root entry; 

Archive CAF, SOFT, and RFT into a single eStream Set; 
Return eStream Set; 

} 

The pseudo-code for the overloaded function ModifyEStreamSet is mentioned below: 

{ 
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// not needed unless merging of uploaded profile data is supported 

} 

Testing design 

This document must have a discussion of how the component is to be tested. 

o Unit testing plans 

The plan for unit testing Package Manager includes the development of a driver 
program. This driver interfaces to the Package Manager and invokes the func- 
tions with different parameters. The list of possible cases is described below: 

1 . Test all interfaces by driving the input parameters with different type of add 
and remove registry values. 

2. Test all interfaces by driving the input parameters by varying numbers of 
spoof and copied files. 

3. Test all interfaces by driving the input parameters with some prefetch infor- 
mation. 

4. Test all interfaces for meaningless input values from the IM and PM. 
o Prefetch block containing file number not assigned by IM. 

o EM assigning non-continguous file numbers. 

5. Test upgrade interface for capability to detect and handle bad eStream Set 
gracefully. 

6. Test upgrade interface and make sure it can detect overlapping file number as- 
signments. 

7. Test upgrade interface and make sure prefetch blocks are not referencing old 
file number from previous versions. 

o Stress testing plans 



o Coverage testing plans 



o Cross-component testing plans 

The output data from the Package Manager is called the eStream Set. This 
eStream Set is the input to a stand-alone test program called the eStream Extrac- 
tor. The Extractor unpacks and 'install' the eStream Set into the local machine 
without an eStream client file system installed. This test is used to quickly verify 
that the eStream Set can be run on a pristine machine. Some of the possible varia- 
tions of the Extractor test includes: 

1 . Non-default system variable names. I.e. %SystemRoot% set to "D:\Win" in- 
stead of ' 'C:\Winnf. 

2. Non-default eStream FS drive letter. Use Y instead of Z. 
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Upgrading/Supportability/Deployment design 

The Package Manager logs all error messages to a predefined file common to all compo- 
nents of the Builder program. Every Builder component prints the error message along 
with its component name. This allows the user of the Builder program to quickly track 
down any problem during the Building of a new eStream Set. 



Open Issues 

o Which Builder component creates the installation DLL when the application 
needs the custom installation code? Is a new component needed to create the cus- 
tom DLL separately and insert into AppInstallBlock in the eStream Set as 
needed? 
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Sanjay Pujare and David Lin 
Version OA 



Functionality 

The eStream Application Builder File Access Monitor (FAM) is a kernel-mode device 
driver that behaves as a file filter driver to has the following responsibilities: 

□ Monitor any running application's request to access a file or directory 

□ Track application file and directory accesses 

□ Track file metadata queries 

□ Start and stop profiling via IOCTL requests from the user-mode program 

□ Return the file access data to the user-mode program via I/O Request Packet 
(IRP) 

a Return any error conditions to the user-mode program via IRP 

The File Access Monitor is based on the 'Filemon' program. The source code for the 
program is available free for download over the Web at 
http://www.sysinternals.com/filemon.htm. 

Data type definitions 

The File Access Monitor (FAM) monitors a sequence of file block accesses by a particu- 
lar process or one of its child processes. The FAM also tracks any queries on the file 
metadata. The combination of the file content and metadata is returned to the Profile 
Manager for further processing. The following is the data structure externally visible to 
the other subcomponents outside FAM. 

Struct SequericeData 

{ 

UINT NumEntries ; 
Struct Entry 

{ 

PUNICODE_STRING FilePathName ; 
BOOL IsAccessingMetadata; 
ULONG Offset; 
ULONG Size; 
} Entries [NumEntries] ; 

}; 

The FileName contains the null terminating string of the file accessed. IsAccessMetadata 
flag indicates if the access is on the file metadata or the file content. If the operation is on 
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the file content, then the fields 'Offset' and 'Size' indicate the location of the read or 
write operations. Otherwise, the fields 'Offset' and 'Size' are not used. 



Interface definitions 



Function 1 : Hooks into user defined IOCTL calls 

// The following is a Fast I/O Device Control 
// call interface. Each of the user IOCTL 
// call to the driver is described here. 
// The OICTL input and output parameters are 
// stored on the IRP on the InputBuffer and 
// OutputBuffer respectively. 

// This function must be called only by NT I/O 
// Manager. 

BOOLEAN FileSysFastloDeviceControl ( 
IN PFILE_OBJECT FileObject, 
IN BOOLEAN Wait, 
IN PVOID InputBuffer, 
IN ULONG InputBufferLength, 
OUT PVOID OutputBuffer, 
IN ULONG OutputBuf ferLength, 
IN ULONG IoControlCode, 
OUT PIO_STATUSJBLOCK IoStatus, 
IN PDEVICE_OBJECT DeviceObj ect ) 

Input : 

I oCon t r o 1 Code = = I OCTL_FAM_VERS I ON 

OutputBuffer: version number of the driver 

I oCon t r o 1 Code = = I 0CTL_FAMJSTART 

InputBuffer: process ID to monitor 



I oCon t r o 1 Code = = 1 0CTL_FAM_STOP 
OutputBuffer: stop profiling 

I oCon t r ol Code = = I OCTL_FAM_GETDATA 
OutputBuffer: get sequence data 

I oCon t rol Code= = I OCTL_FAM_GETSTATUS 

OutputBuffer: get status from driver 



Output : 
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Comments: 

The IOCTL calls from the user program to 
the device driver is either through the 
dispatcher or through the Fast I/O 
interface . 

Errors: 

Function 2: DriverEntry 

// Called by the NT system to initialize 

// driver. The following entries are hooks 

// into the OS and are not called by any of our 

// component directly. 

NTSTATUS DriverEntry ( 

IN PDRIVER_OBJECT DriverObj ect , 
IN PUNICODEJSTRING RegistryPath) 

Comments : 

Initialize the driver 

Function 3: Hooks into Fast I/O functions 

// NT Fast I/O calls. These are some of the 
// hooks into the OS 
NTSTATUS FileSysFastIoRead( 

IN PDRIVER_OBJECT DriverObj ect , 

IN PLARGE_ INTEGER FileOffset, 

IN ULONG Length, 

IN BOOLEAN Wait, 

IN ULONG LockKey, 

OUT PVOID Buffer, 

OUT PIO_STATUS_BLOCK IoStatus, 

IN PDEVICE_OBJECT DeviceObj ect ) 

Comments: 

Hooks into Fast I/O Read 

Function 4: Hooks into dispatcher functions 

// Besides hooks into Fast I/O calls, we 
// must also hook into each of the major 
// functions like IRP_MJ_CREATE, IRP_MJ_READ, 
// etc... 

FileSysHookRoutine ( 

PDEVICE_OBJECT HookDevice, 
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IN IRP Irp) 

Component design 

I/O Hook location 

The trickiest part of the File Access Monitor (FAM) component design is determining the 
locations in the Operating System to hook the routines. FAM must provide driver entry 
function for initializing the driver. It must also provide hooks into the OS to monitor 
read and write file operations. In addition, it needs to monitor access to file metadata. 
And finally, it has to provide the user-mode program a way to communicate to the FAM 
through the IOCTL calls. 

The FAM must behave like any other Windows kernel-mode drivers by exporting the 
standard DriverEntry function. The DriverEntry function has the following purposes: 

o Check for OS build version. 

o Setup the device name 

o Call OS routine to create the device 

o Make symbolic link to allow device access from Win32 programs . 

o Create dispatch points for all routines that must be handled 

o Setup Fast I/O hooks 

o Initialize all data structures 

In addition to the DriverEntry, the FAM must handle five user defined IOCTL calls. 



o 


IOCTL 


FAM 


VERSION 


o 


IOCTL 


FAM 


START 


o 


IOCTL 


FAM 


"stop 


o 


IOCTL 


FAM_ 


"getdata 


o 


IOCTL 


FAM 


"getstatus 



In IOCTLFAMSTART, the handler receives the process ID from the user-mode pro- 
gram. It uses this process ID to filter out relevant file and metadata accesses. In 
IOCTL_FAM_STOP, the handler stops monitoring and recording any file accesses. In 
IOCTLJF AM_GETD AT A , the handler packages the file access sequence in the I/O Re- 
quest Packet (IRP) to be returned to the user-mode program. Finally, in 
IOCTL_FAM_GETSTATUS, the handler returns its current status. This status includes: 
FAM_STATUS_OK, FAM_STATUS JSRROR, and FAM_STATUS ^PROFILING. 

In addition to the user defined IOCTL hooks, the FAM must add hook into both the dis- 
patch points and the Fast I/O calls to monitor all read and write requests. In addition, the 
FAM monitors any metadata accesses. The following is a list of Fast I/O calls it must 
hook: 

o FastloRead 
o FastloWrite 



Omnishift Technologies, Inc. 



4 



Company Confidential 



eStream Builder File Access Monitor Low Level Design 



o FastloMdlReadComplete 

o FastloMdlWriteComplete 

o FastloReadCompressed 

o FastloWriteCompressed 

o FastloQueryBasicInformation 

o FastloQueryStandardlnformation 

In the routine to handle FastloRead and Fastlo Write, the driver must determine the proc- 
ess ID making this request. If the process is in the list of monitoring processes, the file 
name, file offset, and size is recorded and added to the profile sequence list. In the rou- 
tine to handle FastloQueryBasicInformation and FastloQueryStandardlnformation, the 
driver records the file name associated with this metadata query. 

In addition to hooks to the Fast I/O calls, the I/O may call the File System services 
through standard Windows NT dispatch points. The following is a list of dispatch points 
to be handled by FAM: 



o IRP MJ CREATE 

o IRPMJREAD 

o IRPJVIJWRITE 

o IRPJVIJ JDIRECTORY JTONTROL + IRP MN QUERY DIRECTORY 

o IRP MJ QUERY INFORMATION 

o IRP_MJ SET INFORMATION 

o IRP_MJ_QUERY_EA 

o IRP_MJ_SETjEA 

The routine to handle IRPJMJJREAD, IRP JvU_WRITE, and 
IRPJV4NQUERYDIRECTORY is handled by the same function as the routine for 
handling FastloRead and FastloWrite. The routine to handle 
IRP.MJQUERYJDNFORMATION, IRPMJSETJQsfFORMATION, 
IRP_MJ_QUERY_EA, and IRP_MJ_SETJEA are handled by the same function as the 
routine for handling FastloQuerylnformation. 

Communication with user-mode component (Profile Manager) 
Besides using the IOCTL to send profile data to the Profile Manager, the FAM must also 
signal the Profile Manager when new data is available for retrieval. The Profile Manager 
wakes up from the signal by the FAM and retrieves the information on the blocks of files 
accessed. FAM also signals the profile manager when the profiled application termi- 
nates. FAM uses KeSetEventQ to send a 'data available' event signal to the profile man- 
ager. Profile manager calls KeWaitForSingleEventQ or KeWaitForMultipleEventQ to 
wait for a signal from the kernal-mode driver. KeClear Event () is called by the FAM 
when the signal to profile manager should be deactivated. 

Process Filtering 

The FAM must filter the profile information so only relevant data relating to the applica- 
tion under profiled is obtained. This is accomplished by filtering the data according to 
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the process ID invoking the file access operations. When the FAM is started, the Profile 
Manager sends its process ID. FAM assumes all child processes of the Profile Manager 
process ID is to be monitored since the Profile Manager invokes all applications using 
Creat eProcess () API. Thus, the new processes all inherit Profile Manager as its parent 
process ID. The process filtering is accomplished using PsSetCreateProcessNotify- 
RoutineQ to add a hook to the OS. FAM is notified whenever there is a new process cre- 
ated. The process ID is recorded in a list if its ancestor is the Profile Manager process ID. 
This list is used to filter the profile data gathered by FAM. 

Locks 

Since multiple threads may be entering different sections of FAM and accessing different 
data structures, appropriate locks must be used to prevent multiple threads from reading 
and writing at the same time. ExInitalizeResourceLiteQ, ExAcquireResourceExclu- 
siveLiteQ, and ExReleaseResourceLiteQ are used when shared data structure is accessed. 
These APIs have the requirement that the kernel APCs must be disabled before calling 
and that IRQL must be lower than DISPATCHLEVEL. This can be accomplished by 
using KeEnterCriticalRegionQ and KeLeaveCriticalRegionQ. The following is a sample 
code using these APIs: 

ERESOURCE gResource; // global variable 

KeEnterCriticalRegion () 

ExAcquireResourceExclusiveLite (&gResource, TRUE) ; 

<critical section of code> 

ExReleaseResourceLite (kgResource) ; 
KeLeaveCriticalRegionO ; 

Testing design 

o Unit testing plans 

The plan for unit testing of the FAM consists of using the Profile Manager (PM) 
and a File Access Driver (FAD) as the test drivers. The PM tests user-defined 
IOCTL calls. The FAD creates desired data pattern from the OS's I/O Manager 
to the FAM. The FAD tests the FAM's ability to monitor file accesses by query- 
ing files and directories in a particular order. Together, the PM and FAD test 
coverage of the FAM is complete. The following is a list of tests: 

1 . Test each user-defined IOCTL interface via PM by sending border cases. 

2. Test to make sure FAM captures every file and directory access via standard 
file I/O requests from a user-mode program called FAD. 
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o Stress testing plans 



o Coverage testing plans 



o Cross-component testing plans 

Cross-component testing for the Builder program is described in the Package 
Manager low-level design document. 

Upgrading/Supportability/Deployment design 

Other Builder components log error messages to a predefined file. The kernel-mode pro- 
grams do not have the capability to read/write to a file. Since FAM is a kernel-mode 
program, an alternative method of reporting error messages has to be developed. Current, 
the FAM has a user-defined IOCTL interface (IOCTL_FAM_GETSTATUS) to retrieve 
the error messages. FAM keeps a stack of error messages encountered and reports the 
stack of error messages at the request by an appropriate user-mode program. 

Open Issues 

o Exactly which Fast I/O calls need to be hooked to get all the read and write opera- 
tions for file accesses? 

o Along the same line, which dispatch points need to handled to get all the read and 
write operations for file accesses? 

o Have we hooked into all possible places where the metadata accesses can occur? 

o Does the FAM need to hook into FileLock and FileUnlock operations? 
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Functionality 

The eStream Application Builder Profile Manager is responsible for the following: 

□ Receive request from the UI Component for one or more application executables 
to profile. 

□ Accumulate each run of the profile data in a data structure suitable for merging. 

□ Invoke each application executable for a fixed amount of time, for a fixed number 
of prefetch blocks, for a simple start-stop of the program, or for multi-level profil- 
ing based on scripts or manual usage of an application. 

□ Communicate with File Access Monitor (FAM) kernel-mode driver using 
IOCTLs to start and stop profiling. 

□ Obtain the complete file access sequence data from the FAM. 

□ Process the file access sequence into two parts: a table of file access frequency 
and a list of prefetch blocks. 

□ Send the resulting profile data to Package Manager component for integration into 
the AppInstallBlock. 



This component will probably exist as a class object and will be instantiated by the 
Builder user interface component. The component will run in the same process as the 
user interface component. Please see Builder User Interface component document for 
more information on that component. 



Data type definitions 

The Profile Manager imports the file access sequence from the FAM. The data structure 
is described below: (Please see the File Access Monitor for detail information on the 
meaning of each field in the data structure) 



Struct SequenceData 

{ 

UINT NumEntries; 
Struct Entry 

{ 

UNICODE_STRING FilePathName ; 
BOOL IsAccessingMetaData; 
ULONG Offset; 
ULONG Size; 
} Entries [NumEntries] ; 
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}; 

Profile Manager creates two data structure from the data received from the Install Moni- 
tor and the FAM. The consumer of the output data structure is the Package Manager. 
These data structures is described in the following two structures: 

Struct Pref etchBlockList { 
UINT NumSections; 
Struct Pref etchBlocks { 
UINT BlockType; 
UINT NumEntries; 
Struct Entry { 

UNICODE_STRING FilePathName ; 
ULONG BlockNumber; 
} Entries [NumEntries] ; 
} Entries [NumSections] ; 

}; 

Struct Prof ileApplicat ions { 
UINT NumEntries ; 
Struct Entry { 

UNICODE_STRING FilePathName; 

UNICODEJSTRING Arguments ; 
} Entries [NumEntries] ; 

}; 

The access count is used to order the list of files in a directory according to metadata file 
access frequency. The prefetch data is encorporated into the AppInstallBlock by the 
Package Manager to be used by the eStream Client. 

Interface definitions 
Function 1 : StartProfiling 

int StartProfiling ( 

IN PPROFILE_APPLICATIONS AppList, 

IN UINT Type, 

IN UINT TypeData, 

OUT PPREFETCH_BLOCKS Pref etchBlocks) 
Input: 

AppList: a list of file pathnames and 
Arguments to run the application. 

Type: type of profiling to do 

SIMPLE: start and stop application 
TIMEBASED: profile for a fixed time and 
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terminate application 
SIZEBASED : profile for a fixed size of 
Profile data 

TypeData: extra data for profile type 
If Prof ileType==SIMPLE, TypeData is 
Ignored 

If Prof ileType—TIMEBASED, TypeData is 

Length of time in seconds 
If Prof ileType—SIZEBASED, TypeData is 

Size of profile data in bytes 
If Prof ileType==COMMANDBASED / TypeData is 

Pointer to a possible list of script 

Files to be invoked 

Pref etchBlocks : List of prefetch blocks 
To add to the AppInstallBlock 

Return value: 

Success or failure code of the profiling 

Comments: 

The profile manager component actually send 
IOCTLs to the file filter device driver to 
Start and stop the gathering of the profile 
Data and to retrieve the profile data. 

Errors : 

FileNotFound: some of the application 

Executables in the list may not exist or 

Not readable 
Prof ileTimeout : failed to gather the desired 

Size of the profile data after certain 

Amoun t of t i me 
DriverFailure : File Access Monitor return 

Failure code to the Profile Manager which 

Must propagate the error to the user 

Interface 

Component design 

Application Invocation 

To start profiling, the component must have a list of application pathname to be invoked. 
The pathname may be an executable with a list of arguments. Or the pathname may be a 
Windows short-cut file. If the pathname is an exe file, then the standard Win32 API Cre- 
ateProcessQ is used. On the other hand, if the file is a Windows short-cut file, then the 
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component needs to extract relevant information from the short-cut file to make the 
proper CreateProcessQ invocation. The following is a pseudo-code for extracting path- 
name and argument information from the short-cut file using IShellLink interface: 

GetlnfoFromShortCutFile (char *strPath, char *strArq) 
{ 

I shell Link *psl ; 
WIN32_FIND_DATA fd; 

if (SUCCEEDED ( CoCreatelstance (CLSID_ShellLink, 
NULL, 

CLSCTX_INPROC_SERVER , 
IID_IShellLink, 
(LPVOID*) &psl))) 

{ 

psl->GetPath(strPath, MAX_LEN, &fd, 0) ; 
Psl - >Get Arguments (strArg, MAX_LEN) ; 
psl->Release () ; 

} 

} 

Command-based Profiling 

One of the options for profiling includes the ability to identify blocks of files accessed 
when specific application command is invoked. The profiler prompts the user for the de- 
sired actions (ie. Open document, save document, etc) and gathers file blocks that are ac- 
cessed corresponding to those actions. These commands are saved into the Applnstall- 
Block for eStream client to intelligently pick the proper set of blocks to stream. The fol- 
lowing is an enumeration of some divisions of prefetch blocks: 

o Start Application 
o End Application 
o Save Document 
o Open Document 
o Cut Sections 
o Copy Sections 
o Paste Sections 

In the current design, the profiler divides the code into two prefetch sections. The first 
section contains the critical blocks necessary for efficient startup of the streamed applica- 
tion for the very first time. The blocks of code in this section include starting of applica- 
tion. The second section is the common blocks necessary for efficient running of the 
streamed application. This includes common user operations like opening and saving of 
document. 

Communication with kernel-mode driver (FAM) 

The Profile Manager communicates with the kernel-mode driver to retrieve the informa- 
tion on the blocks of files accessed. The profile manager waits for a signal from the 
FAM indicating a new data is available for retrieval. FAM also signals the profile man- 
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ager when the profiled application terminates. FAM uses KeSetEventQ to send a 'data 
available" event signal to the profile manager. Profile manager calls KeWail- 
ForSingleEventQ or KeWaitForMultipleEventQ to wait for a signal from the kernal-mode 
driver. KeClear Event () is called by the FAM when the signal to profile manager should 
be deactivated. 

Pseudo-code 

The pseudo-code for the function StartProfiling is described below: 
{ 

Initialize GlobalFileAccessCounts and GlobalPrefetchBlocks; 

Load FAM if not loaded; 

For each (executable in the list of application) { 

Start FAM by sending it process ID of the Profile Manager; 

Create new process for this executable and run it; 

Switch (Type of Profiling) { 

Case SIMPLE: 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} Until application start up; 
Break; 
Case TIMEBASED: 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} Until fixed time unit; 
Break; 
Case SIZEBASED: 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} while (size < fixed amount); 
Break; 

Case COMMANDBASED: 
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For each command in the list { 
If (script exist) 

Run script on the executable program; 
Else 

Prompt operator for proper action; 
Loop { 

Wait for an event from FAM; 
Get Status from FAM; 
Get SequenceData from FAM; 
} Until script completed; 

} 

} 

Send WM_QUIT message to the application process; 
Loop { 

Wait for an event from FAM; 

Get Status from FAM; 

Get SequenceData from FAM; 
} Until application quit; 
Inform FAM to stop gathering profile data; 

Compute PrefetchB locks from the SequenceData and append to 
GlobalPrefetchBlocks; 

> 

Unload File Access Monitor (if possible); 
Return GlobalPrefetchBlocks; 

} 

Testing design 

o Unit testing plans 

The plan for unit testing the Profile Manager includes developing a driver to con- 
nect to the interface between the Profile Manager and the Builder UI. The driver 
conducts the following types of tests: 

1 . Test different type of profiling including simple profiling, time-based profil- 
ing, size-based profiling, and script-based profiling. 
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2. Test different executable programs and make sure the output data "makes 
sense". 

3. Test a list of executables for merging capability of the Profile Manager. 

4. Test the interface between Profile Manager (PM) and the File Access Monitor 
(FAM) using FAM as the test driver. The FAM can check for any valid 
IOCTL calls from the PM. FAM can also reply to IOCTL calls with different 
values in the IRP to simulate all possible cases. 

o Stress testing plans 



o Coverage testing plans 



o Cross-component testing plans 

Cross-component testing for the Builder program is described in the Package 
Manager low-level design document. 

Upgrading/Supportability/Deployment design 

The Profile Manager logs all error messages to a predefined file common to all compo- 
nents of the Builder program. Every Builder component prints the error message along 
with its component name. This allows the user of the Builder program to quickly track 
down any problem during the Building of a new eStream Set. 

Open Issues 

o How to automate profiling so the application doesn't require any user interven- 
tion? Look into using Rational TestSuite programs. 

o Can a subset of Winstone be used for profiling? How do we determine which part 
of the profile data is more useful to the end-user? 

o Should Profile Manager actually create data structures like PrefetchBlocks that 
require FileNumber assignments? Or should Profile Manager just create the out- 
put data without knowning FileNumbers? Then Package Manager associates the 
file numbers assigned by the Install Monitor with the profile data gathered by the 
Profiler. 
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Nicholas Ryan 
Version 1.5 




Functionality 

The Application Install Manager (AIM) is a component of the eStream client executable. 
It is responsible for installing and uninstalling eStream applications at the request of the 
License Subscription Manager (LSM). AIM uses the information contained in an Appln- 
stallBlock to prepare the user's system for execution of a given eStream application. It 
creates registry entries, copies files, and updates the file spoofing database. The user can 
then launch his application via a local shortcut or a shortcut on the eStream drive. Unin- 
stallation involves undoing all changes made to the user's system by AIM during installa- 
tion. 

Data type definitions 

This component uses the AppInstallBlock, but doesn't define it. This is defined in a low- 
level design document for the Builder component. 

The AppInstallBlock is a binary data file with a versioned interface, basically consisting 
of: 

• a header 

• a list of files to install or send to the file spoofer 

• a list of registry entries to install or remove 

• a set of prefetch requests to communicate to the profile/prefetch component 

• a set of initial profile data to communicate to the profile/prefetch component 
(post- version 1.0) . 

• a comment section 

• an embedded DLL that can be loaded and executed for custom install needs 

• a section containing a license agreement to be shown to the user 

Many of the AIMsc functions take an AIBFileRef as an argument, which is an opaque 
pointer to a Builder-provided class called 'AppInstallBlock'. This utility class will be 
shared by the Builder and the AIM code. It encapsulates the complexity of reading from 
and writing to the various sections of an AppInstallBlock file. 

Also, each application has a prefetch data file created for it an install time that is initial- 
ized with prefetch data from the AppInstallBlock. This data file is named and located as 
described in the Component Design section, and consists of a list of critical blocks fol- 
lowed by a list of common blocks. The file begins with the following structure: 
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typedef struct 
{ 

OTUInt32 numCriticalBlocks / 
OTUInt32 numCommonBlocks ; 

} Pref etchFileHeader, *pPref etchFileHeader ; 

The remained of the file consists of a non-padded list of the following structures: 

typedef struct 
{ 

OTUI nt 3 2 f i 1 eNumber ; 
OTUI nt 3 2 bl ockNumber ; 

} Pref etchltem, *pPrefetchItem; 

The following data types are used in the AIM and AIMsc interfaces: 

typedef void *AIBFileRef; 

Error codes are defined in the OTError enumeration, which lives in a file called 
ot_errors.h. 

Interface definitions 
Application installation/uninstallation 

There are only two functions exposed by AIM, one for application installation, and an- 
other for application uninstallation. Only the License Subscription Manager will be call- 
ing these functions, 

J 

OTError 

AIMInstallApplication(OTUInt8 appld[16], const char *pathToAIB) 
Parameters 

appld 

[in] The application ID of the eStream application to install. 
pathToAIB 

[in] Pointer to a null-terminated string that specifies a path to an Appln- 
stallBlock file to install 

Return Values 
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OTSUCCESS if all the actions specified in the AppInstallBlock were performed 
successfully, an error code otherwise. 

Comments 

None. 

OTError 

AIMUninstallApplication(OTUlnt8 appld[16]) 
Parameters 

appld 

[in] The application ID of an existing eStream application to uninstall. 
Return Values 

If the specified application ID is not recognized, or the original AppInstallBlock 
is not found, an error code will be returned. Otherwise, AIM will make an attempt 
to undo all of the actions it performed while installing this application. It will re- 
turn OT SUCCESS if it undid enough of these actions so that any future installa- 
tion of the same application will succeed. 

Comments 

None. 

AIM Sub-Component Interface 

Much of the functionality required by the AIM design will be useful to the Builder testing 
framework as well. This functionality will be treated as a sub-component within the AIM 
component, called AIMsc, and will export a well-defined interface. That interface is de- 
fined as follows. 

OTError 

AIMscOpenApplnstallBlock(constchar *pathToAIB, AlBFileRef *pAIBFile) 
Parameters 

pathToAIB 

[in] Pointer to a null-terminated string that specifies a path to an AppIn- 
stallBlock file to open. 

pAIBFile 
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[out] Returns a reference to an open ApplnstallBlock file. 
Return Values 

OT_SUCCESS if the ApplnstallBlock was opened successfully and validated, an 
error code otherwise. 

Comments 

The reference returned by this function can be used as a parameter to any of the 
other functions that take an AIBFileRef. 

OTError 

AIMscCloseApplnstallBlock(AIBFileRef aibFile) 
Parameters 

. aibFile 

[in] An opaque reference to an open ApplnstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

Return Values 

OT_SUCCESS if the close succeeded, an error code otherwise. 
Comments 
None. 

void 

AIMscGetAIBAppldfAIBFileRef aibFile, OTUInt8 pAIBAppld[1 6]) 
Parameters 

aibFile 

[in] An opaque reference to an open ApplnstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

pAIBVersion 

[out] Returns the value of the Appld field in the ApplnstallBlock. 
Return Values 
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OT SUCCESS if the value was successfully retrieved, an error code otherwise. 
Comments 
None. 

void 

AIMscGetAIBVersionNo(AlBFileRef aibFile, OTUInt32 *pAIBVersionNo) 
Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AMscOpenAppInstallBlock. 

pAIB VersionNo 

[out] Returns the value of the VersionNo field in the AppInstallBlock. 
Return Values 

OT SUCCESS if the value was successfully retrieved, an error code otherwise. 
Comments 
None. 

void 

AIMscGetAIBShouldRebootf 
AlBFileRef aibFile, 
OTBool *pAIBShouldReboot) 

Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AMscOpenAppInstallBlock. 

pAIBShouldReboot 

[out] Returns the value of the ShouIdReboot flag in the AppInstallBlock. 

Return Values 
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SUCCESS if the value was successfully retrieved, an error code otherwise. 
Comments 

None. 
OTError 

AIMscGetAIBAppName( 
AlBFileRef aibFile, 
char *pAIBAppName, 
OTUInt16 *pSizeAIBAppName) 

Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

pAIBAppName 

[out] The value of the ApplicationName field in the AppInstallBlock is 
copied into the memory pointed to by this address (it will be null termi- 
nated). 

pSizeAIBAppName 

[in, out] On input, should point to the size of the memory at pAIBApp- 
Name. On output, will point to the total bytes needed to hold the entire 
string if OTERRORJBUFFER_TOO_SMALL is returned, otherwise is 
undefined. 

Return Values 

OT_SUCCESS if the value was successfully retrieved, OTER- 
RORBUFFERTOOSMALL if the buffer is too small to hold the entire string, 
or another error code otherwise. 

Comments 

None. 

OTError 

AIMscCheckAIBCompatibleOS(AIBFileRef aibFile) 
Parameters 
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aibFile 



[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 



Return Values 



OTSUCCESS if the OS version was successfully retrieved and if the currently 
running OS is compatible. 



Comments 



This function will check if the currently installed operating system and service is 
compatible with the specified AppInstallBlock (using the compatibility informa- 
tion contained in the AppInstallBlock). If not, it will return OTER- 
ROR_INCOMPATIBLE_OS. 



OTError 

AIMsclnstallAppFiles( 
AlBFileRef aibFile, 
HKEY spoofKey, 
const char *installLogFile, 
OTBool *plsRebootNeeded) 



Parameters 

aibFile 



[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

spoofKey 

[in] An open handle to the registry key where file-spoofing data is stored. 
installLogFile 

[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 

phRebootNeeded 

[out] Returns TRUE if a reboot is needed to complete the file copying, 
FALSE otherwise. 



Return Values 



Omnishift Technologies, Inc. 



7 



Company Confidential 



eStream <COMPONENT> Low Level Design 



OT_SUCCESS if all file install operations succeeded, an error code otherwise. 
Comments 

This function will perform the file copies and add the file spoofing entries speci- 
fied in the File section of the AppInstallBlock. Changes will be appended to the 
install log file (it is created if it doesn't already exist). 

This function will not undo file copies or spoof entry additions if it fails. 
AIMscUninstaJIAppFiles should be called to do so. 

OTError 

AIMscUnins tallAppFiles( 
AlBFileRef aibFile, 
HKEY spoofKey, 
const char *installLogFile, 
OTBool *plsRebootNeeded) 

Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

spoofKey 

[in] An open handle to the registry key where file-spoofing data is stored. 
installLogFile 

[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 

phRebootNeeded 

[out] Returns TRUE if a reboot is needed to complete the file deletions, 
FALSE otherwise. 



Return Values 



OTSUCCESS if enough of the file install operations were reversed so that re- 
installation will succeed and so that the system is in a consistent state. Otherwise, 
an error code is returned. 



Comments 
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This function will reverse the file additions and remove the file spoof database en- 
tries specified in the install log file. 

OTError 

A IMsclns tall A pp Variables( 
AlBFileRef aibFile, 
const char *installLogFile, 
cons t char *varRefCountFile) 

Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

installLogFile 

[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 

varRefCoun tFile 

[in] A null-terminated string representing the path to the variable recount- 
ing file for this user. 

Return Values 

OT SUCCESS if all variable modifications succeeded, an error code otherwise. 
Comments 

This function will perform the add/remove variable (i.e. registry entry) changes 
specified in the Variable section of the AppInstallBlock. Changes will be ap- 
pended to the install log file (it is created if it doesn't already exist). 

This function will not undo registry modifications if it fails. AIMscUninstal- 
lAppVariables should be called to do so. 

OTError 

AIMscUninstallAppVariables( 
AlBFileRef aibFile, 
const char *installLogFile, 
const char *varRefCountFile) 

Parameters 
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aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock. 

installLogFile 

[in] A null-terminated string representing the path to a text file to which 
change entries should be added. 

varRefCountFile 

[in] A null-terminated string representing the path to the variable recount- 
ing file for this user. 

Return Values 

OT_SUCCESS if enough of the variable changes were reversed so that re- 
installation will succeed and so that the registry is in a consistent state. Otherwise, 
an error code is returned. 

Comments 

This function will reverse the add/remove variable (i.e. registry entry) changes 
specified in the install log file. 

OTError 

AIMsclnstallAppPrefetchFi!e(AIBFileRef aibFile, const char *pref etch File) 
Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

prefetchFile 

[in] A null-terminated string representing the path to the prefetch file to be 
created. 

Return Values 

OTSUCCESS if prefetch block installation succeeded, an error code otherwise. 
Comments 
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This function will install the prefetch information contained in the Prefetch sec- 
tion of the AppInstallBlock into prefetchFile. 

OTError 

AIMscUninstallAppPrefetchFile(AIBFileRefaibFile, const char *pre fetch File) 
Parameters 

AIBFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

pPrefetchFile 

[in] A null-terminated string representing the path to the prefetch file to be 
uninstalled. 

Return Values 

OT SUCCESS if prefetch block uninstallation succeeded, an error code other- 
wise. 

Comments 

This function will remove the prefetch information stored at prefetchFile. 
OTError 

AIMscCallCustomlnstall(AIBFileRefaibFile) 
Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

Return Values 

An error code if the custom code .dll could not be extracted, loaded and called. 
Otherwise, returns the value returned by the InstallQ function in the custom code 
.dlh 

Comments 
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This function will extract and load the custom code .dll included in the Code sec- 
tion of the AppInstallBlock, and then call the exported .dll function named In- 
stallQ. 

OTError 

AIMscCallCustomUninstall(AIBFileRefaibFile) 
Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

Return Values 

An error code if the custom code .dll could not be extracted, loaded and called. 
Otherwise, returns the value returned by the UninstallQ function in the custom 
code .dll. 

Comments 

This function will extract and load the custom code .dll included in the Code sec- 
tion of the AppInstallBlock, and then call the exported .dll function named 
UninstallQ. 

OTError 

AIMscEnforceLicenseAgreement(AIBFileRef aibFile) 
Parameters 

aibFile 

[in] An opaque reference to an open AppInstallBlock previously returned 
by AIMscOpenAppInstallBlock 

Return Values 

OTSUCCESS if the license agreement was successfully displayed and agreed to 
by the user, OTERROR JJSER_CANCELLED, if it was displayed and not agreed 
to by the user, or an error code if it could not be displayed. 

Comments 

This function will extract the license agreement text included in the LicenseA- 
greement section of the AppInstallBlock and display it to the user. The user will 
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be given the option to agree or not agree to the license (probably via a pair of but- 
tons in a dialog). OTERRORUSERCANCELLED is returned if the users de- 
cides not to accept the license agreement. 

OTError 

AIMscDisplayCommen t(AIBFileRef aibFile) 
Parameters 

aibFile 

[in] An opaque reference to an open ApplnstallBlock previously returned 
by AIMscOpenAppInstallBlock 

Return Values 

OT SUCCESS if the comment was successfully displayed, an error code other- 
wise. 

Comments 

This function will display to the user the comment included in the Comment sec- 
tion of the ApplnstallBlock. 



Component design 

AIMsc does not have hard-coded knowledge regarding any of the standard registry and 
file locations used by AIM, which is why the functions in its interface take as inputs 
specifiers for filenames and base registry locations. Conversely, AIM itself has no 
knowledge of the internal structure of the ApplnstallBlock file, which is why it must call 
AIMsc functions to work with such files. 

Expansion is perform on registry entries and file paths containing certain variables, when 
they are read from the ApplnstallBlock. These variables are defined in the Builder-LLD 
and will be recognized and expanded by AIM. (This includes file-spoof entries.) 

AIM stores its data in the expected places for an eStream client component. All of the 
data AIM stores is user-specific, so it makes no use of the global locations defined for 
eStreams. 

Registry keys 

AIM stores its registry keys and values under: 
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HKEY_CURRENT_USER\SOFTWARE\Omnishift\eStream\AIM 



This key will have its permissions modified so that ordinary users cannot modify the key 
(but the eStream client service will be given privileges so that it can do so). Here are the 
subkeys AIM places under this key: 

"SpoofEntries" 

Spoof entries are placed here. All spoofing is done globally, so there is no need to place it 
under an eStream-app specific key. Each value under this key is a pair of pathnames as 
follows: 

<old-pathname> (REGMULTISZ) 
- <array of spoofed-pathnames> 

The array contains one pathname for each installed eStream app that is spoofing the file. 
The file spoofer currently only spoofs to one of these files at a time, but this corresponds 
to a kind of refcounting of the spoof entry. 



Every installed eStream app has its own subkey whose name is a string representation of 
its Appld, like so: "{00000000-0000-0000-0000-00000000000}". The values stored un- 
der each such key are: 

Appld (REGJBINARY) 

- Appld in binary form (16 bytes) 
AppName (REG_SZ) 

- name of the application (same as in the AppInstallBlock) 
AppInstallBlockPath (REGJSZ) 

- path to the AppInstallBlock for the application 
AppInstallState (REG DWORD) 

- a value of 0 means app is installed, 1 means install is in progress, 2 mean 
uninstall is in progress. 



AIM stores per-user files at the following path: 

(Path to the user's home directory)\Application Data\Omnishift\eStream\AIM 
Here is what is stored in this directory: 



<AppId>' 



Files 



RegistryRefCounts.dat 



- registry refcounts for eStream apps 
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<AppId GUID string> - a folder for each installed app 

RegistryRefCounts.dat is a file that associates a refcount with every registry key that is 
added and every value that is modified during an eStream app install. Each null- 
terminated line in the file is of the form: 

<hash of registry keyWalue pathxtab><refcount of registry key\value> 

For each installed application, a separate data folder is created. The name of the folder is 
the Appld of the application in GUID ASCII format, like so: "{00000000-0000-0000- 
0000-00000000000}". The files stored under each such folder are: 

<GUID string>-AIB.dat - the AppInstallBlock file for the application 

<GUID string>-Prefetch.dat - the prefetch data file for the application 
InstallLog.txt - a generated log of what to do during uninstall 

The Prefetch data file is simply an array of Prefetchltem structures (as described in the 
Data Structures section). 

The InstallLog.txt is a list of undoable actions taken during installation. This log will be 
used during uninstall to determine which files and entries are safe to remove. Each line in 
the file contains one change, and is of the form: 

ADDED or OVERWROTE or SPOOFED FILE <filename> (fully qualified) 
ADDED or OVERWROTE KEY <keyname> (fully qualified) 
ADDED or OVERWROTE VALUE <valuename> (fully qualified) 

AIMInstallAppIication Prototype 

Installing an eStream application consists of the following steps: 
Preparing for the installation 

Displaying a license agreement to the user and having him agree to it 
Installing all required local files and spoof entries for this app 
Setting/removing registry entries as required 
Initializing the prefetch data for this app 
Performing any required custom installation tasks 
Displaying the comment to the user if required 
Completing the installation 
Rebooting the computer if necessary 

AIM's policy is that if it encounters any fatal error during the execution of AIMInstal- 
lAppIication, it will attempt to undo everything it did before returning. AIM also grace- 
fully handles aborted installs and uninstalls. 



1. 
2. 
3. 
4. 
5. 
6. 
7. 
8. 
9. 
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Step 1 - Preparing for the installation 

First, AIM checks if the application is already installed by looking for an Appld registry 
key for the specified Appld. If found, then the AppInstalllnProgress registry value is 
checked. If it exists and is I , the user is asked if he wants to re-install, otherwise, he is 
asked to restart an aborted or damaged installation. If the user says no, AIMInstallAp- 
plication cleans up and exits with OTERRORUSERCANCELLED. 

Next, a free disk space check is performed to ensure that enough disk space is available 
for the install. The available free space must be at least twice the size of the Applnstall- 
Block to proceed. If not, AIMInstallApplication exits with OTER- 
ROR_VOLUME_FULL. 

Next, an Appld folder is created for the app (described earlier), and the AppInstallBlock 
file is copied to this folder. Then Appld registry key is created and the four defined val- 
ues created and initialized. The AppInstallState value in particular is set to 1 to indicate 
an install is in progress. Then, AIM then opens the AppInstallBlock using AIMscO- 
penAppInstallBlock, and calls AIMscCbeckAIBCompatibleOS to check for OS com- 
patibility. If any of these operations fail, AIMInstallApplication cleans up and exits with 
an error. 

Step 2 - Displaying the license 

AIMscEnforceLicenseAgreement is called to display the license text to the user and ask 
for his agreement. If the functions fails or if the user rejects the agreement, AIMInstal- 
lApplication cleans up and exits with OTERROR USER CANCELLED. 

Step 3 - Installing local files 

The install log file to be used for this application is created or open and truncated. AIM- 
scInstallAppFiles is called to copy the install files to the computer and to create the 
spoof entries specified in the AppInstallBlock If the function fails, AIMInstallApplica- 
tion cleans up and exits with an error. 

If it succeeds, a boolean is returned indicating whether a reboot needs to occur due to 
shared files being overwritten. This value is remembered for use in step 9. 

Step 4 - Modifying the registry 

AIMscInstallAppVariables is called to perform the registry modifications specified in 
the AppInstallBlock. If the function fails, AIMInstallApplication cleans up and exits 
with an error. 
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Step 5 - Initializing prefetch data 

AIMscInstallAppPrefetchFile is called to create and initialize the prefetch file for this 
application. If the function fails, AIMInstallApplication cleans up and exits with an er- 
ror. 

Step 6 - Performing custom install tasks 

AIMscCallCustomlnstall is called to extract the custom code .dll contained in the Ap- 
plnstallBlock and to call the InstallQ function it exports. If AIMscCallCustomlnstall 
fails, AIMInstallApplication cleans up and exits with an error. 

Step 7 - Displaying a comment 

AIMscDisplayComment is called to display any comment to the user contained in the 
appropriate section of the Applnstallblock. If this function fails, AIMInstallApplication 
cleans up and exits with an error. 

Step 8 - Completing the installation 

The AppInstalllnProgress registry value is set to 0 to indicate the install is complete. 
AIMscOpen AppInstallBlock is called to close the AIBFileRef opened in step 1 , and any 
handles to open registry keys are also closed. 

Step 9 - Rebooting the computer (if necessary) 

If AIMscInstallAppFiles in step 3 returned a value indicating a user reboot is necessary, 
or if AIMscGetAIBShouldReboot is called and returns a value of TRUE, the user is 
asked to reboot. Otherwise, no reboot is performed and the application is ready to be run. 
AIMInstallApplication exits returning OT SUCCESS. 

AIMUninstallAppIication Prototype 

Uninstalling an eStream application consists of the following steps: 

1 . Preparing for the uninstallation 

2. Undoing all additions made to the registry during install 

3. Removing all files copied during install and removing spoof entries for this app 

4. Deleting the prefetch data for this application 

5. Performing any required custom uninstallation tasks 

6. Completing the uninstallation 

7. Rebooting the computer if necessary 

If the uninstallation fails for any reason, the called should tell the user that the uninstall 
has failed and that he should attempt to re-install the application before trying to uninstall 
again. 
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Step 1 - Preparing for the uninstallation 

First, AIM checks if the application is already installed by looking for the Appld registry 
key corresponding to the specified Appld. If not found, then AIMUninstallApplication 
exits with OTERRORJTEMNOTFOUND. 

Then, the AppInstallState value is set to 2 to indicate an uninstall is in progress. AIM- 
scOpenAppInstallBlock is called to open the AppInstallBlock at the path specified by 
the AppInstallBlockPath key. If this fails, then AIMUninstallApplication exits with an 
error. 

Step 2 - Undoing registry modifications 

AIMscUninstallAppVariables is called to reverse the registry additions made during 
installation. If the function fails, uninstall cannot proceed safely and AIMUninstallAp- 
plication exits with an error. 

Step 3 - Undoing file copies and removing spoof entries 

AIMscUninstallAppFiles is called to delete the files copied during install and to remove 
the spoof entries written then. If the function fails, uninstall cannot proceed safely and 
AIMUninstallApplication exits with an error. 

If it succeeds, a boolean is returned indicating whether a reboot needs to occur due to 
shared files being overwritten. This value is remembered for use in step 7. 

Step 4 - Deleting profile/prefetch data 

AIMscUninstallAppPrefetchFile will be called to remove the prefetch data stored for 
this application. Any failure is ignored. 

Step 5 - Performing custom uninstall tasks 

AIMscCallCustomUninstall is called to extract the custom code .dll contained in the 
AppInstallBlock and call the UninstallQ function it exports. If AIMscCallCustomUnin- 
stall fails, uninstall cannot proceed safely and AIMUninstallApplication exits with an 
error. 

Step 6 - Completing the uninstallation 

AIMscGetAIBShouIdReboot is called and the return value saved. Then AIMscO- 
pen AppInstallBlock is called to close the AIBFileRef opened in step 1, and the Appld 
folder and all its contents are deleted. The Appld registry key and all its subkeys are de- 
leted also. Any handles to open registry keys are closed. Any failures here are ignored. 
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Step 7 - Rebooting the computer (if necessary) 

If AIMscUninstallAppFiles in step 3 returned a value indicating a user reboot is neces- 
sary, or if AIMscGetAIBShouIdReboot is called and returns a value of TRUE, the user 
is asked to reboot. Otherwise, the uninstallation is complete. AIMUninstallApplication 
exits returning SUCCESS. 

AIMsc Function Prototypes 

Prototypes for the AIMsc functions declared earlier are given in this section. 
OTError 

AIMscOpenAppInstaIIBIock(const char *pathToAIB, AIBFileRef *pAIBFile) 

A class AppInstallBlock object is created, and the appropriate member function is called 
to open and verify the AppInstallBlock. An opaque pointer to this object is returned in the 
pAIBFile parameter. 

OTError 

AIMscCloseAppInstallBlock(AIBFileRef aibFile) 

The AppInstallBlock class object pointed to by aibFile is deleted, 
void 

AIMscGetAIBAppId(AIBFHeRef aibFile, OTUInt8 pAIBAppId|16]) 
void 

AIMscGetAIBVersionNo(AIBFileRef aibFile, OTUInt32 *pAIBVersionNo) 
void 

AIMscGetAIBShouldReboot( 
AIBFileRef aibFile, 
OTBool *pAIBShouldReboot) 

OTError 

AIMscGetAIBAppName( 
AIBFileRef aibFile, 
const char *pAIBAppName, 
OTUIntl6 * P SizeAIBAppName) 

These four functions are trivial. They directly map to method calls on the AppInstall- 
Block class object pointed to by aibFile, which retrieves the associated header field of the 
AppInstallBlock. (See the interface declaration for AJMscGetAIBAppName for details 
on its calling logic.) 
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OTError 

AIMscCheckAIBCompatibleOS(AIBFileRef aibFile) 

This function will call an API such as GetVersionEx (for Windows) to determine the cur- 
rently running operating system. The OS version is then compared to the OS version 
stored in the AppInstallBlock by calling a method on the AppInstallBlock object pointed 
to by aibFile. 

OTError 

AIMscInstallAppFiles( 
AIBFileRef aibFile, 
HKEY spoofKey, 
coDSt char *instal!LogFiIe, 
OTBool *pIsRebootNeeded) 

The appropriate methods in the AppInstallBlock object pointed to by aibFile are called to 
retrieve the retrieve the entries in the File section of the AppInstallBlock. 

The File section is organized as a series of trees, with directories as non-leaf nodes and 
plain files as leaf nodes. All nodes are stored contiguously according to the pre-order tra- 
versal of the trees. The ABVIsc code does not parse the section itself, but relies upon the 
code in the provided AppInstallBlock class to do the unpacking for it. 

For every file copied or spoof entry added by the algorithm, an entry is made to the file at 
InstallLogFile. 

The algorithm is as follows: 

while there are file entries to read in the File section 
read an entry 

if entry filename contains Builder/AIM defined variables 
replace variables with local expansions 

if the file node is a spoof entry 
call HandleSpoofEntryO 

else 

call HandleCopyEntryO 

Here is HandleSpoofEntryO: 

if filename already exists 

if existing version is earlier or version can't be read 
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mark for spoofing 
else // filename does not exist 

create zero-length file at filename 
mark for spoofing 

if marked for spoofing 

create/modify spoof entry under SpoofKey 

Here is HandleCopyEntry(): 

if filename already exists 

increment shared file ref count in registry 

if existing version is earlier or version can't be read 
mark for copy 
else // filename does not exist 

mark for copy 

if marked for copy 

attempt to copy node file to client computer 
if copy fails 

tell system to perform copy at reboot 

The phRebootNeeded argument will be set to TRUE if any file copies were scheduled to 
happen at reboot, FALSE otherwise. Additionally, if any spoof entries were added, then 
an IOCTL will be sent to the spoof driver asking it to reload the spoof database. 

The shared file reference count mentioned in the algorithm above is stored in a standard 
place in the Windows registry. AIM will create or increment this reference count for 
every non-spoofed file included in the AppInstallBlock (they can all be potentially shared 
since they will be placed outside of the eStream app directory). Each such file has an as- 
sociated REG DWORD value under the key at: 

HKEY^LOCAL^MACHrNE\SOFTWARE\Microsoft\Windows\CurrentVersion\ 
SharedDLLs 

Even though the subkey name is called 'SharedDLLs', this key is used by all well- 
behaved applications to refcount all shared files. The value's name is the path to the file 
and the value's data is a integer that is the reference count for this file. 

OTError 

AIMscUninstallAppFiles( 
AIBFileRef aibFile, 
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HKEY 
const char 
OTBool 



spoofKey, 

*installLogFile, 

*pIsRebootNeeded) 



Currently, the aibFile parameter is not even needed since all of the information needed 
for file uninstall is contained in the log file at InstallLogFile. However, it is included to 
enforce a design decision, which is that the user should have a valid reference to an Ap- 
plnstallBlock before performing any install/uninstall related actions on an eStream app. 

The algorithm for AIMscUninstallAppFiles is simple. It iterates over the change entries 
contained in the log file, and undoes file copies and spoof entry additions when it is safe 
to do so. Here is the algorithm: 

while there are change entries in the log file 
read the next entry 

if the entry is of the form "ADDED or OVERWROTE <filename>" 
decrease refcount of file 
if refcount is 0 

attempt to delete file 
if deletion fails 

tell system to perform deletion at reboot 

else if the entry is of the form "SPOOFED <filename>" 
decrease refcount of spoof entry for this filename 
if refcount is 0 

delete/modify the spoof entry 
if 0-byte placeholder at filename still exists 
delete it 

A failure to delete a file or to schedule its deletion will not cause AIMscUninstal- 
lAppFiles to fail. 

OTError 

AIMscInstallAppVariables( 
AIBFileRef aibFile, 
const char *installLogFile, 
const char *varRefCountFile) 

The appropriate methods in the AppInstallBlock object pointed to by aibFile are called to 
retrieve the retrieve the entries in the Variables section of the AppInstallBlock. 
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The Variable section is organized as a series of trees, similar to how the File section is 
organized. There are two types of nodes, key nodes and value nodes. Registry keys can 
contain other keys and registry values, while registry values are just name/data pairs that 
are stored in keys. The AIMsc code does not parse the section itself, but relies upon the 
code in the provided AppInstallBlock class to do the unpacking for it. 

Each entry can be either an add key/value entry, or a delete key/value entry. For every 
change made, an entry is made to the file at installLogFile. Also, registry key and value 
additions made by eStream are refcounted in the file at varRefCountFile, but deletions are 
not refcounted. (Only deletions during uninstall are refcounted). 

The algorithm is as follows: 

while there are variable entries to read in the Variable section 
read an entry 

if entry name contains Builder/ AIM defined variables 
replace variables with local expansions 

if the variable node is a key entry 

call HandleKeyEntryQ 
else 

call Handle VaIueEntry() 

Here is HandleKeyEntry(): 

if the key name is not fully qualified 
error 

if this is an add key entry 

if key doesn't already exist 
create key 
else // it's a delete key entry 

delete the key 
increment key refcount 

Here is HandleCopyEntry(): 

if this is an add value entry 

add the value and its data 

increment value refcount 
else // it's a delete value entry 
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delete the value 

UINT32 
OTError 

AIMscUninstaIlAppVariables( 
AIBFileRef aibFile, 
const char *installLogFiIe, 
const char *varRefCountFiJe) 

Currently, the aibFile parameter is not even needed since all of the information needed 
for file uninstall is contained in the log file atlnstallLogFile. However, it is included to 
enforce a design decision, which is that the user should have a valid reference to an Ap- 
plnstallBlock before performing any install/uninstall related actions on an eStream app. 

The algorithm for AIMscUninstallAppVariables iterates over the change entries con- 
tained in the log file, and undoes registry key and value additions when it is safe to do so. 
The biggest complication is that it must handle COM entries specially, which requires 
two passes. Here is the algorithm: 

Pass 1: 

while there are change entries in the log file 
read the next entry 

if this is an entry of the form "ADDED/OVERWROTE KEY <CLSID 
keyname>" 

check the refcount of the executable that provides this CLSID 
if the refcount is > 0 

mark this CLSID as sacred 

Pass 2: 

while there are change entries in the log file 
read the next entry 

if the entry is of the form "ADDED/OVERWROTE KEY <keyname>" 
decrement keyname refcount 
if keyname refcount is 0 

if keyname is not a COM key with a sacred CLSID value 
delete it and all its subkeys and values 
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else if the entry is of the form "ADDED/OVERWROTE VALUE <value- 
name>" 

decrement valuename refcount 
if valuename refcount is 0 

if value is not part of a COM key with a sacred CLSID value 
delete it 

A failure to delete one or more registry entries will not cause AIMscUninstallAppFHes 
to fail. 

OTError 

AIMscInstallAppPrefetchFile(AIBFileRef aibFHe, const char *prefetchFHe) 

The appropriate methods in the AppInstallBlock object pointed to by aibFHe are called to 
retrieve the data in the Prefetch section of the AppInstallBlock. The data are written out 
into the file atprefetchFile as an array of Prefetchltem structures. Any existing file at 
PrefetchFile is overwritten. 

• Next, the Prefetch component is called to set up an association between the new applica- 
tion and its prefetch file. 

OTError 

AIMscUninstallAppPrefetchFile(AIBFileRef aibFHe, const char *prefetchFile) 

The file at PrefetchFile is deleted. (No call need be made to the Prefetch component.) 
OTError 

AIMscCallCustomInstall(AIBFileRefaibFile) 

The appropriate methods in the AppInstallBlock object pointed to by aibFHe are called to 
retrieve the data in the Code section of the AppInstallBlock. This data is written out as a 
.dll library. This library is loaded and the JnstallQ function export is called (and its return 
value returned). 

OTError y 
AIMscCallCustomUninstall(AIBFileRef aibFHe) 

The appropriate methods in the AppInstallBlock object pointed to by aibFHe are called to 
retrieve the data in the Code section of the AppInstallBlock. This data is written out as a 
.dll library. This library is loaded and the UninstallQ function export is called (and its re- 
turn value returned). 

OTError 

AIMscEnforceLicenseAgreement(AIBFileRef aibFHe) 
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The appropriate methods in the AppInstallBlock object pointed to by aibFile are called to 
retrieve the text in the License Agreement section of the AppInstallBlock. The text is dis- 
played to the user in a dialog box with two buttons, 'Agree' and 'Disagree'. 

OTError 

AIMscDisplayComment(AIBFiIeRef aibFile) 

The appropriate methods in the AppInstallBlock object pointed to by aibFile are called to 
retrieve the text in the Comment section of the AppInstallBlock. The text is displayed to 
the user in a dialog box. 

Testing design 
Unit testing plans 

AIM will be tested by a program that generates AppInstallBlocks designed to stress the 
component. AIM will be asked to install the given AD3 and if successful, the resulting 
state of the system will be compared to the expected state had all the files and variables 
been installed correctly. An uninstall will then be performed and the system state also 
checked. 

The focus of the testing will obviously be on the File and Variable sections. The other 
sections such as Code and Comments will be stressed also, but their boundary conditions 
are much simpler. 

AIM'S ability to gracefully handle aborted installs and uninstalls will also be tested. 

Stress testing plans 

The program described above can be deliberately tuned to create AppInstallBlocks of un- 
usual size and organization. For example, AppInstallBlocks with thousands of files and 
registry entries, or files and entries with unusually long names, etc. 

Coverage testing plans 

In addition to the stress testing, deliberately malformed AppInstallBlocks will be gener- 
ated by the test program to hit as much error-handling code as possible. AIM's data files 
and registry entries can also be deliberately mangled to help achieve this effect. 

Cross-component testing plans 

As soon as they are available, Builder-generated AppInstallBlocks will be tested to verify 
that the AIM is compatible with the Builder's output. As soon as the LSM and a browser 
plugin are available, the communication path from browser to LSM to AIM will be 
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tested. As soon as the file spoofer is available, compatibility with the file spoof entries 
that AIM makes will be tested. 

Upgrading/Supportability/Deployment design 

AIM will make use of the eStream logging facility to record information about errors and 
other unusual conditions that occur. The log file will be useful for diagnosing problems 
that occur during testing and in real world situations. 

If the AIM component is upgraded, it must still be able to uninstall any eStream applica- 
tions installed at the time of upgrade. This entails being able to interpret old AIM registry 
entries and data files, including the AppInstallBlock. This is more a concern for future 
designers of the AIM component, however. 

Open Issues 
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Functionality 

The Client Installer/Uninstaller [C1N] software is used to setup/remove a client's capabil- 
ity to eStream applications via installing/uninstalling eStream client software. [eStream 
client software patches are handled via special encapsulated executables distributed with 
patches & are not discussed herein.] eStrearn client software consists of: 

• kernel-mode drivers [EFSD, FileSpoofer, NoCluster] 

• user-mode modules comprising the privileged eStream Client Executable [ECE]: 

o Client EStream Startup/Shutdown [CES] 

o Estream Prefetch Fetch [EPF] 

o License Subscription Manager [LSM] 

o Application Install Manager [AIM] 

o EStream Cache Manager [ECM] 

o Client Network Interface [CNI] 

o Client User Interface [CUI] 

• user-mode non-privileged executable eSUser.exe, which is run by an eStream user 
at log in, at log out, & optionally at other times 

• user-mode non-privileged Client Browser Module [CBM] called eSCBM.exe, 
which fields notification messages from ASP servers [Design reviewers sug- 
gested investigating combining eSUser.exe & eSCBM.exe, migrating CUI into 
the module, & allowing MIME messages to control UI operations] 

The Component Design section of the document first addresses CIN. 

The Client EStream Startup/Shutdown [CES] software comprises ECE's main; it calls 
client components' initialization routines, it orchestrates activating various eStream client 
drivers, and it handles shutting the eStream client software down. The Component De- 
sign section of the document next addresses CES. 

To understand the proposed operation of the CIN and CES, the reader is expected to be 
familiar with all client LLD design material. 

Data Definitions 

CIN is a standalone process and does not share any data definitions with other portions of 
eStream; registry keys used for communicating data are defined in the Component De- 
sign section. CES obtains, verifies, tracks, and communicates information about the 
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eStream user (his windows account name and a handle for his display) for passing to 
LSMandCUI. 

Interface Definitions 

Registry keys and IOCTL interfaces are described in the Component Design section. Be- 
low are the procedural interfaces. 

From CES: 

• CNIInitialize(void) 

• EPFInitialize(void) 

• ECMInitialize(void) 

• CUIInitialize(IN HandleToDesktop) 

• LSMInitialize(IN WindowsUserName) 

• LSMUpdateAllSubscriptionStatus(void) 

To CES: 

• CESSetMode(IN ModeRequested) - Modes are STANDBY, READY, ACTIVE 

Component Design 

Client Installer/Uninstaller [CIN] 

This section outlines the operations involved in installation & uninstallation of the four 
client packages listed above, after describing special attributes of the ECE and CBM that 
are relevant to CIN. [BTW, to put our installation into perspective, SoftwareWow in- 
stalls 1 9 files: 4 EXEs, 1 DLL, 6 SYSs, 2 VXDs, 4 WAVs, and 2 HLP/CNTs.] 

Attributes of eStream Client Executable 

The ECE is chiefly a performance-critical extension of the eStream client file system and 
certain parts of ECE have been identified as potentially moving to kernel-mode for per- 
formance reasons, including ECM, EPF, LSM and possibly CNL The ECE contains cer- 
tain administrative modules for which it is desirable to execute in the context of a sepa- 
rate privileged account, including AIM, LSM, and possibly CES. Several client tasks 
[eSUser.exe and eSCBM.exe] need to communicate with ECE asynchronously. Given 
this set of characteristics. ECE will be implemented as a daemon process launched at 
boot time. 

On W2K/WNT, ECE is planned to be a windows service program [ref: MSDN Library 
\Platform SDKYWindows Base Services\Executables\Services]. ECE starts running when 
the system boots and it runs under a privileged account to allow it to hide various data for 
privacy and piracy protection. ECM is not a COM server; it does not contain any inter- 
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faces intended to be exported for running as instances inside a COM client. It is also not 
a COM client; it does not manage compound documents or import functionality from any 
COM servers. ECE contains two message queues into which client tasks can post mes- 
sages. One message queue [CBMmessages] handles messages from eSCBM.exe & the 
other message queue [USRmessages] handles messages from eSUser.exe. [Guidance on 
selecting between Clipboard, COM, DDE, FileMapping, Mailslot, Pipes, RPC, Sockets, 
& WM COPYDATA for IPC implementation is found in MSDN Library \Platform SDK 
\Windows Base Services Mnterprocess Communication\Choosing an IPC Mechanism.] 

Attributes of Client Browser Module 

The CBM is a standard browser helper application, i.e., it is a Windows32 application 
that is registered to handle a particular MIME type embedded in HTML. Helper apps are 
supported in both Internet Explorer 4+ and Netscape Navigator 4+. To register a helper 
application for handling a certain MIME type, one sets up the following: 

• key for mime extension under HKLM\MIME\Database\Content Type 

• key under HKLM\SOFTWARE\Classes for class type which application handles 

• key under HKCR for file extension which application handles 

[BTW, Software Wow creates these entries to register a helper app to handle .wow files.] 
CBM talks with ECE by placing messages in the CBMmessage message queue. 
eStream Client Installation 

A standard installation tool [likely InstallShield] will be used to create the installation 
package. Installation images for distribution on CD-ROM and via the internet will be 
provided. Uninstallation via the "Add/Remove Programs" will be supported. 

BTW, the windows installer is discussed in MSDNLibrary/ Platform SDK/ Management 
Services/ Setup/Windows Installer. The windows installer is a privileged service. A 
"managed application" is an application that is installed on a per-machine (not per-user) 
basis & which requires elevated privileges for installation. The application installation is 
marked via the ALLUSERS property as to its installation privilege needs. If app is per- 
machine, but user is unprivileged, installation will fail unless the AlwaysInstallElevated 
registry key is set. 

Installation will be performed under administrative privileges. To handle the situation of 
residual partial installation due to a failed/aborted previous installation, there is always an 
[invisible] uninstall-like step that cleans up before an install. eStream client software 
(and supported eStream applications) expect a client's DLLs to be at a certain revision 
level; the installation process will check them & if they are not at that level or higher, the 
installation will include upgrading them. EStream-specific installation activities are de- 
scribed in the paragraphs that follow. 
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The user is asked where he would like to install eStream-related files, with the default 
location set to C:\Program Files\Omnishift\eStream. The installation path chosen is writ- 
ten to the registry key HKLM\Software\Omnishift\eStream\InstalIPath. A directory for 
TEMP files is created at InstallPath\temp and the temp path is written to the registry key 
HKLM\Soflware\Omnishift\eStream\TempPath. The estream version number is written 
to HKLM\Software\Omnishift\eStream\Version. The user is asked the drive letter/UNC 
path at which to mount the eStream file system & this is put into the registry at 
HKLM\Software\Omnishift\eStream\FileMountLocation. 

The kernel-mode drivers (NoCluster, EFSD, FileSpoofer) are installed in the system di- 
rectory & set up via registry entries for automatic loading at boot time. 

• Copy {NoCluster.sys,efsd.sys,fspoof sys} to %SystemRoot%\system32\drivers 

• Set up registry to specify loading at boot time 

o Create key under HKLM\System\CurrentControlSet\Services w/ values: 

• Set TYPE to SERVICE KERNEL DRIVER or SER- 
VICE JFILE_SYSTEMJ3RTVER (as appropriate) 

• Set START to SERVICESYSTEMSTART 

• Set ERRORCONTROL to SERVICEERRORNORMAL 

Please note that the names NoCluster.sys & fspoof sys are used above for the reader's 
convenience. For the product, we may obscure their purpose via naming them something 
like OTI001 .sys & OTI002.sys. 

The ECE is installed as a windows service & set up for automatic loading at boot time. 

• Create a special privileged account under which the ECE service will run 

• Copy estream.exe to %SystemRoot%\system32\estream.exe 

• Set up registry to specify load at boot time, specify to run under privileged accnt 

o Create key under HKLM\System\CurrentControlSet\Services w/ values: 

• Set TYPE to SERVICE JWIN32J3WNPROCESS 

• Set START to SERVICE_AUTO_START 

• Set ERRORCONTROL to SERVICE JERROR_NORMAL 

• Set IMAGEPATH to %SystemRoot%\system32\estream.exe 

eSUser.exe is copied to location in HKLM\Software\Omnishift\eStream\InstallPath. 
eSCBM.exe is copied to location in HKLM\Software\Omnishift\eStream\InstallPath. 
eStream user-specific actions are set up: 

• During installation, user is asked if he/she would like eStream to be automatically 
activated at log in to windows (preferred). The answer determines which value 
(ready|activate) is sent to eSUser.exe when it is run at the user's login. 
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• User account is set up to run "eSUser.exe ready|activate" whenever user logs in to 
windows. 

• User gets a Start\Program link which runs "eSUser.exe activate''. (BTW, both 
SoftwareWow & NewMoon have StartVProgram entries.) 

• User account is set up to run "eSUser.exe standby" whenever user logs out of 
windows. 

• Registry keys are set up to hold user's ASP Data Set and Subscription Data Set. 
eStream Client Uninstallation 

eStream-specific installation activities are undone at uninstall; the files NoCluster.sys, 
efsd.sys, fspoof.sys, estream.exe, eSUser.exe, & eSCBM.exe are deleted, the relevant 
registry keys are removed, and the user-specific operations for each eStream user on this 
client are yanked. We need to decide if apps should be [automatically] uninstalled when 
user requests that eStream client software be uninstalled. 

Client eStream Startup/Shutdown [CES] 

The actions of CES need to be understood in the context of the overall operation of the 
eStream client software; hence, the description of CES below is embedded in a general 
discussion of the operation of ECE. This is followed by a section considering the issues 
surrounding usability when eStream client software is in READY (i.e., not yet active) 
mode for a user. 

Operation of ECE 

At system boot time, the kernel-mode drivers are loaded, but they are not performing 
eStream-related activities. ECE is loaded as a service program belonging to the eStream 
privileged account set up at install time and is ready to receive messages in its message 
queues. The eStream client software is considered to be in STANDBY mode. No 
systray icon is displayed. 

When an eStream user logs in to the client's console, "eSUser.exe ready|activate" is run 
and it sends a READY message to ECE on behalf of this user. [Please note that a client 
system only supports one active eStream user at a time.] When ECE receives the mes- 
sage, it: 

• runs initialization routines for each client module that has them [expected order 
from lowest to highest level: CNIInitialize, LSMInitialize, EPFInitialize, ECMIni- 
tialize, CUIInitialize] 

• calls the LSM interface that reads in this user's ASP & Subscription data [stored 
in privileged registry entries] and that performs a synchronize operation 

• brings up a systray icon on the eStream user's desktop 
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The eStream client is now considered to be in READY mode on behalf of this user; it 
remains in READY mode if eSUser.exe was invoked with the ready parameter. 

Whenever eSUser.exe is run with the activate parameter (either automatically at the 
user's windows login or manually via a Start/Programs entry selection), it sends an AC- 
TIVATE message to ECE on behalf of the user. [A discussion of issues related to 
eStream being activated automatically in other situations is included below.] The 
eStream client software moves from READY mode to ACTIVE mode for the user in 
question, performing the following actions: 

• CES sends lOCTLs to the kernel-mode drivers, signaling them to begin eStream- 
related activities: 

o NoCluster: CES sends IOCTL to activate [or may be active at boot] 

o EFSD: CES sends IOCTL_EFS_START_FS 

o FileSpoofer: CES sends IOCTL_FS_START_SPOOFING 

• If any ECE routines need to execute separate initialization on client activation, 
those routines are called at this point. 

When eStream client software is in ACTIVE mode for a user, that user can seamlessly 
run eStream applications; when it is in READY mode for a user, eStream applications 
cannot be executed immediately. Issues involving eStream client software automatically 
transitioning from READY to ACTIVE when a user attempts to run an eStream applica- 
tion or providing a meaningful error message to the user if that automatic transition is not 
supported are considered in the next section. 

ECE may be deactivated automatically at eStream user logout [eSUser.exe standby] or 
manually on demand via the CUI (mode request for READY). When the ECE is deacti- 
vated, the following events are performed: 

• CES sends IOCTLs to the kernel-mode drivers, signaling them to stop eStream- 
related activities: 

o NoCluster: CES sends IOCTL to deactivate [or may continue active] 

o EFSD: CES sends IOCTL_EFS_SHUTDOWN_FS 

o FileSpoofer: CES sends IOCTL_FS_STOP_SPOOFING 

• If any ECE routines need to execute separate termination on client deactivation, 
those routines are called at this point. 

ECE brings up a browser to provide web-based ABOUT/HELP information for eStream 
and to allow the user to interact with his ASP to obtain/modify account-specific data. For 
eStream 1 .0, this communication is handled via the standard mechanism used by cur- 
rently-available windows application that provides web access, i.e., the user's default 
browser is invoked in a separate window with a particular URL and set of parameters 
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[decided at 9/7 meeting of Lacky, Bhaven, Manoj, and me]. A common way to accom- 
plish this is to execute a variant of the ShellExecute command. 

Automatic Transition from READY to ACTIVE mode Plus Error Handling 

If the eStream client software is not active when an eStream user tries to execute an 
eStream application, it is attractive to either give a meaningful error message or to auto- 
matically activate eStream. [To my knowledge, doing either is an eFSD issue; I am in- 
cluding a discussion of the issue here since the topic seems to be open at this time.] To 
give a meaningful error message, eFSD needs to see the relevant file reference, meaning 
that either the eStream drive letter needs to exist or eFSD needs to be registered as a 
UNC provider; otherwise, we may give a somewhat inscrutable error message. To auto- 
matically transition from READY to ACTIVE when we see a relevant file reference, we 
need to ensure that file spoofing is not needed prior to eFSD receiving the reference to 
the inactive eStream file system for apps of interest. Please note that it is not required 
that eStream client software be activated automatically if CBM finds it inactive. 

Testing Design 

Unit and Coverage testing plans 

For ON, an installation/deinstallatidn of stub versions of the various eStream client soft- 
ware will be developed. For CES, a test harness will be developed to test all CES opera- 
tional modes (standby, transition to ready, transition to active, transition to inactive, shut- 
down) and is expected to achieve essentially 100% PFA coverage. 

Stress testing plans 

CIN stress testing will involve testing a myriad of software and hardware system configu- 
rations. CES stress testing will involve testing behavior in the presence of various prob- 
lems with the eStream client software configuration and for various user situations. 

Cross-component testing plans 

CIN is expected to be incorporated with the first set of working eStream 1 .0 software & 
installing/uninstalling should form a key step in the daily automated testing of eStream 
1 .0 going forward. CES is also expected to be included in the first set of working 
eStream 1 .0 software and should also get a daily workout. 

Upgrading/Supportability/Deployment Design 

Both CIN and CES should be developed with an eye to minimizing issues related to up- 
grading/supportability/deployment. The bar is very high for these components to contain 
heavy error checking and reporting. 
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Attachment 1 : eStream Client Mode Transitions 



Current Mode 



Mode Request 



Normal Mode Transition 



• Standby 

• Standby 

• Ready 

• Ready 

• Active 

• Active 



Ready 

Active 

Active 

Standby 

Ready 

Standby 



Transition to Ready 

Transition to Ready, then to Active 

Transition to Active 

Transition to Standby 

Transition to Ready 

Transition to Ready, then Standby 



Attachment 2: Running scripts at Logon & at Logout 

Windows2000 supports executing scripts & programs at user logon & logoff Registry 
keys control whether the executions are synchronous or asynchronous wrt logon or logoff 
& whether the processes interact with the user. To install these processes, one runs start/ 
run/gpedit.msc & chooses UserConfigurationAVindowsSettings/Scripts(Logon/Logoff). 



Attachment 3: eStream Client IPC 



ASPNotify sends ASPUserName, SLiMServerSet in MIME to eSCBM; ASPPromptNotify 
posts message to CGI script on ASPWebServerName asking for ASPNotify for QOS problem 





CBMMessageQueue: ASPU- 

serName,SLiMServerSet, 

ASPWebServerName 



USRMessageQueue: Mode, 
WindowsUserName, Handle- 
ToDesktop 




User login executes with ready or activate parameter; user logout executes with standby pa- 
rameter; user can request deactivate from CUI inside ECE 
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Introduction 

The Client Installer/Uninstaller [ON] software is used to setup/remove a client's 
capability to eStream applications via installing/uninstalling eStream client software. 
[eStream client software patches are handled via special encapsulated executables 
distributed with patches & are not discussed herein.] eStream client software consists of: 

• kernel-mode drivers [EFSD, FileSpoofer, NoCluster] 

• user-mode modules comprising the privileged eStream Client Executable [ECE]: 

o Client EStream Startup/Shutdown [CES] 

o Estream Prefetch Fetch [EPF] 

o License Subscription Manager [LSM] 

o Application Install Manager [AIM] 

o EStream Cache Manager [ECM] 

o Client Network Interface [CNI] 

o Client User Interface [CUI] 

• user-mode non-privileged executable eSLogin.exe, which is run by an eStream 
user at log in, at log out, & optionally at other times 

• user-mode non-privileged Client Browser Module [CBM] called eSCBM.exe, 
which fields notification messages from ASP servers 

This document first addresses ON. 

The Client EStream Startup/Shutdown [CES] software comprises ECE's main; it calls 
client components' initialization routines, it orchestrates activating various eStream client 
drivers, and it handles shutting the eStream client software down. This document next 
addresses CES. 

To understand the proposed operation of the CIN and CES, the reader is expected to be 
familiar with all client LLD design material. 

Client Installer/Uninstaller [CIN] 

This section outlines the operations involved in installation & uninstallation of the four 
client packages listed above, after describing special attributes of the ECE and CBM that 
are relevant to CIN. [BTW, to put our installation into perspective, SoftwareWow 
installs 19 files: 4 EXEs, 1 DLL, 6 SYSs, 2 VXDs, 4 WAVs, and 2 HLP/CNTs.] 

Attributes of eStream Client Executable 

The ECE is chiefly a perfonnance-critical extension of the eStream client file system and 
certain parts of ECE have been identified as potentially moving to kernel-mode for 
performance reasons, including ECM, EPF, LSM and possibly CNI. The ECE contains 
certain administrative modules for which it is desirable to execute in the context of a 



separate privileged account, including AIM, LSM, and possibly CES. Several client 
tasks [eSLogin.exe and eSCBM.exe] need to communicate with ECE asynchronously. 
Given this set of characteristics. ECE will be implemented as a daemon process launched 
at boot time. 

On W2K/WNT, ECE is planned to be a windows service program [ref: MSDN Library 
\Platform SDKAWindows Base Services\Executables\Services]. ECE starts running when 
the system boots and it runs under a privileged account to allow it to hide various data for 
privacy and piracy protection. ECM is not a COM server; it does not contain any 
interfaces intended to be exported for running as instances inside a COM client. It is also 
not a COM client; it does not manage compound documents or import functionality from 
any COM servers. ECE contains two message queues into which client tasks can post 
messages. One message queue [CBMmessages] handles messages from eSCBM.exe & 
the other message queue [LGNmessages] handles messages from eSLogin.exe. 
[Guidance on selecting between Clipboard, COM, DDE, FileMapping, Mailslot, Pipes, 
RPC, Sockets, & WMCOPYDATA for IPC implementation is found in MSDN Library 
\Platform SDK \Windows Base Services Ymterprocess Communication\Choosing an PC 
Mechanism.] 

Attributes of Client Browser Module 

The CBM is a standard browser helper application, i.e., it is a Windows32 application 
that is registered to handle a particular MIME type embedded in HTML. Helper apps are 
supported in both Internet Explorer 4+ and Netscape Navigator 4+. To register a helper 
application for handling a certain MIME type, one sets up the following: 

• key for mime extension under HKLM\MIME\Database\Content Type 

• key under HKLM\SOFTWARE\Classes for class type which application handles 

• key under HKCR for fde extension which application handles 

[BTW, Software Wow creates these entries to register a helper app to handle .wow files.] 
CBM talks with ECE by placing messages in the CBMmessage message queue. 

eStream Client Installation 

A standard installation tool [likely InstallShield] will be used to create the installation 
package. Installation images for distribution on CD-ROM and via the internet will be 
provided. Uninstallation via the "Add/Remove Programs" will be supported. 

Installation will be performed under administrative privileges. To handle the situation of 
residual partial installation due to a failed/aborted previous installation, there is always an 
[invisible] uninstall-like step that cleans up before an install. eStream client software 
(and supported eStream applications) expect a client's DLLs to be at a certain revision 
level; the installation process will check them & if they are not at that level or higher, the 
installation will include upgrading them. eStream-specific installation activities are ' 
described in the paragraphs that follow. 



The user is asked where he would like to install eStream-related files, with the default 
location set to C:\Program Files\Omnishift\eStreaml.O. The installation path chosen is 
written to the registry key HKLM\Software\Omnishift\eStreamL0\InstallPath. A 
directory for TEMP files is created at InstallPath\temp and the temp path is written to the 
registry key HKLM\Software\Omnishift\eStreaml.0\TempPath. The estream version 
number is written to HKLM\Software\Omnishift\eStreaml.0\eStream Version. 

The kernel-mode drivers (NoCluster, EFSD, FileSpoofer) are installed in the system 
directory & set up via registry entries for automatic loading at boot time. 

• Copy {NoCluster.sys,efsd.sys,fspoof sys} to %SystemRoot%\system32\drivers 

• Set up registry to specify loading at boot time 

o Create key under HKLM\System\CurrentControlSet\Services w/ values: 

• Set TYPE to SERVICE KERNEL DRTVER or 
SERVICEFILESYSTEMDRJVER (as appropriate) 

• Set START to SERVICEJSYSTEMJSTART 

• Set ERRORCONTROL to SERVICEERRORNORMAL 
Please note that the names NoCluster.sys and fspoofsys are used above as a convenience 
to the reader. For the product, we may obscure their functionality via naming them 
something like OTIOOl.sys and OTI002.sys. 

The ECE is installed as a windows service & set up for automatic loading at boot time. 

• Create a special privileged account under which the ECE service will run 

• Copy estream.exe to %SystemRoot%\system32\estream.exe 

• Set up registry to specify load at boot time, specify to run under privileged accnt 

o Create key under HKLM\System\CurrentControlSet\Services w/ values: 

• Set TYPE to SERVICE JWIN32J3WN_PROCESS 

• Set START to SERVICE_AUTO_START 

• Set ERRORCONTROL to SERVICE ERROR NORMAL 

• Set IMAGEPATH to %SystemRoot%\system32\estream.exe 

eSLogin.exe is copied to location in HKLM\Software\Omnishift\eStreaml.0\InstallPath. 
eSCBM.exe is copied to location in HKLM\Software\Omnishift\eStreaml.0\InstallPath. 

eStream user-specific actions are set up: 

• During installation, user is asked if he/she would like eStream to be automatically 
activated at log in to windows (preferred). The answer determines which value 
(readyjactivate) is sent to eSLogin.exe when it is run at the user's login. 

• User account is set up to run "eSLogin.exe ready|activate" whenever user logs in 
to windows. 

• User gets a StartVProgram link which runs "eSLogin.exe activate". (BTW, both 
SofhvareWow & NewMoon have StartXProgram entries.) 

• User account is set up to run "eSLogin.exe deactivate'' whenever user logs out of 
windows. 

• Registry keys are set up to hold user's ASP Data Set and Subscription Data Set. 



eStream Client Uninstallation 



eStream-specific installation activities are undone at uninstall; the files NoCluster sys 
efsd.sys, fspoofisys, estream.exe, eSLogin.exe, & eSCBM.exe are deleted, the relevant 
registry keys are removed, and the user-specific operations for each eStream user on this 
client are yanked. 

Client eStream Startup/Shutdown [CES] 

The actions of CES need to be understood in the context of the overall operation of the 
eStream client software; hence, the description of CES below is embedded in a general 
discussion of the operation of ECE. This is followed by a section considering the issues 
surrounding usability when eStream client software is in READY (i.e not yet active) 
mode for a user. ' 



Operation of ECE 

At system boot time, the kernel-mode drivers are loaded, but they are not performing 
eStream-related activities. ECE is loaded as a service program belonging to the eStream 
privileged account set up at install time and is ready to receive messages in its message 
queues. The eStream client software is considered to be in STANDBY mode No 
systray icon is displayed. 

When an eStream user logs in to the client's console, "eSLogin.exe readylactivate" is run 
and it sends a READY message to ECE on behalf of this user. [Please note that a client 
system only supports one active eStream user at a time.] When ECE receives the 
message, it: 

• runs initialization routines for each client module that has them [expected order 
from lowest to highest level: CNIStartup, LSMStartup, EPFStartup, ECMStartup 
CUIStartup] r 

• calls the LSM interface that reads in this user's ASP & Subscription data [stored 
in privileged registry entries] and that performs a synchronize operation 

• brings up a systray icon on the eStream user's desktop 

The eStream client is now considered to be in READY mode on behalf of this user- it 
remains m READY mode if eSLogin.exe was invoked with the ready parameter. ' 

Whenever eSLogin.exe is run with the activate parameter (either automatically at the 
a S ^t J A ^ t I OWS l0gm ° r manua,, y via a Start/Programs entry selection), it sends an 
ACTIVATE message to ECE on behalf of the user. [A discussion of issues related to 
eStream being activated automatically in other situations is included below ] The 
eStream client software moves from READY mode to ACTIVE mode for the user in 
question, performing the following actions: 

• CES sends IOCTLs to the kernel-mode drivers, signaling them to begin eStream- 
related activities: 

o NoCluster: CES sends IOCTL to activate [or may be active at boot] 



o EFSD: CES sends IOCTL_EFS_START_FS 

o FileSpoofer: CES sends IOCTL_FS_START_SPOOFING 

• If any ECE routines need to execute separate initialization on client activation, 
those routines are called at this point. 

When eStream client software is in ACTIVE mode for a user, that user can seamlessly 
run eStream applications; when it is in READY mode for a user, eStream applications 
cannot be executed immediately. Issues involving eStream client software automatically 
transitioning from READY to ACTIVE when a user attempts to run an eStream 
application or providing a meaningful error message to the user if that automatic 
transition is not supported are considered in the next section. 

ECE may be deactivated automatically at eStream user logout [eSLogin.exe deactivate] 
or manually on demand via the CUI. When the ECE is deactivated, the following events 
are performed: 

• CES sends IOCTLs to the kernel-mode drivers, signaling them to stop eStream- 
related activities: 

o NoCluster: CES sends IOCTL to deactivate [or may continue active] 

o EFSD: CES sends IOCTL_EFS_SHUTDOWNFS 

o FileSpoofer: CES sends IOCTL_FS_STOP_SPOOFING 

• If any ECE routines need to execute separate termination on client deactivation, 
those routines are called at this point. 

ECE brings up a browser to provide web-based ABOUT/HELP information for eStream 
and to allow the user to interact with his ASP to obtain/modify account-specific data. For 
eStream 1.0, this communication is handled via the standard mechanism used by 
currently-available windows application that provides web access, i.e., the user's default 
browser is invoked in a separate window with a particular URL and set of parameters 
[decided at 9/7 meeting of Lacky, Bhaven, Manoj, and me]. A common way to 
accomplish this is to execute a variant of the ShellExecute command. 

Automatic Transition from READY to ACTIVE mode Plus Error Handling 

If the eStream client software is not active when an eStream user tries to execute an 
eStream application, it is attractive to either give a meaningful error message or to 
automatically activate eStream. [To my knowledge, doing either is an eFSD issue; I am 
including a discussion of the issue here since the topic seems to be open at this time.] 
To give a meaningful error message, eFSD needs to see the relevant file reference, 
meaning that either the eStream drive letter needs to exist or eFSD needs to be registered 
as a UNC provider; otherwise, we may give a somewhat inscrutable error message. To 
automatically transition from READY to ACTIVE when we see a relevant file reference, 
we need to ensure that file spoofing is not needed prior to eFSD receiving the reference to 
the inactive eStream file system for apps of interest. Please note that it is not required 
that eStream client software be activated automatically if CBM finds it inactive. 
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Functionality 

The Client Network Interface (CNI) provides the interfaces for sending messages to serv- 
ers and provides threads for receiving responses and dispatching them appropriately. It 
uses the eStream Messaging Service (EMS) APIs to send and receive various messages to 
and from the application servers and SUM servers. 

The number of threads in the CNI will depend on the functionality available from the 
EMS. In particular, more threads are necessary if the EMS provides asynchronous mes- 
saging capability (and the CNI uses this interface). The interfaces presented by the CNI 
are identical for both cases, but the internal organization of the component is not. 

The prefetcher will make calls to client networking interfaces (indirectly through ECM- 
ReservePage) to send requests for pages. Similarly, the LSM will make calls to acquire 
access tokens and subscription information. 

The networking component is responsible for examining the stream of requests to it and 
deciding when to coalesce multiple page requests into a single request to the server. 

The EMS does not provide reliability in the event of server failure. The CNI is responsi- 
ble for handling server failover and reissuing failed requests on different servers. The 
CNI abstracts the servers from other parts of the system. Clients of the CNI don't need to 
specify a particular server to make a request. 

Since the client networking component is where timeouts and retries occur, it is the com- 
ponent that controls the policies for how long we wait for a connection to time out and 
how many times we retry a request before giving up. These parameters will be tunable. 
Any other parameters of the CNI that make sense to tune will be tunable. 

The CNI is also the component responsible for implementing the server selection policy. 

Data type definitions 

The CNI uses the request structure defined by the ECM. 

The CNI maintains an internal queue of messages that must be sent to servers. This 
queue is not exposed outside of the CNI. Like the ECM request queue, this queue will be 
maintained as a circular, doubly-linked list. 
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typedef struct _NWRequest 
NWRequestType type; 

union {} parameters; /* params, depends on type */ 
struct _NWRequest *next; 
struct JNWRequest *prev; 
} NWRequest; 

typedef enum 
{ 

CNI_PAGE_READ, 
CNI_ACQUIRE_ACCESS_TOKEN , 
CNIJ3ET_I^TEST_APP_INFO, 
CNIJRENEW_ACCESS_TOKEN, 
CNI_RELEASE_ACCESS_TOKEN , 
CNI_REFRESH_APP_SERVER_SET , 
CNI_GET_SUBSCRIPTION_LIST 
} NWRequestType; 

Jhe CNI provides an enumeration of the parameters that can be tuned. This enumeration 
is expected to grow as the number of tunable parameters grows. 

typedef enum 

{ 

CNI_NUM_RETRIES , 
CNI_TIMEOUT, 
CNI_PROXY_ADDRESS , 
CNI_EFFECTIVE_BANDWIDTH 
} NWTunableParameter; 

Related Components 

The prefetcher and LSM call on the CNI to send requests to the app and SLiM servers. 
The CNI makes calls to the ECM and LSM to inform them of responses that have come 
back from the server. The CNI will also make calls to EFSD interface functions when 
pages come back that satisfy EFSD requests. 

Interface definitions 

CNIGetPage 
eStreamStatus CNIGetPagef 

IN ApplicationID app, 

IN EStreamPageNumber page 

); 

CNIGetPage is the interface used by the ECM function ECMReservePage to request 
that a page be sent by the server. (ECMReservePage is called indirectly by the pre- 
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fetcher.) Note that no distinction is made between prefetches and demand fetches. To 
prevent race conditions or deadlock, the requested pages must already be marked as "in 
flight" in the index, and any requests for these pages from the EFSD must already be on 
the "in flight" queue before calling this interface. 

The CNI is responsible for selecting a server to direct this request to, and resending in the 
event of network or server failure. It will coalesce requests for multiple pages from the 
same application into a single request to the server. 

CNIGetSubscriptionList 
eStreamStatus CNIGetSubscriptionList( 

IN string Username, 

IN string Password 

); 

CNIGetSubscriptionList enqueues a request to acquire a subscription list from a SLiM 
server. When the subscription list is returned by the server, the client response thread 
will notify the LSM of the returned data via a callback defined in the LSM document. 

CNIGetLatestApplicationlnfo 
eStreamStatus CNIGetLatestApp!icationInfo( 
IN uin tl 28 SubscriptionID 

); 

CNIGetLatestApplication enqueues a request to get the latest application information 
for a particular app. When the server returns the result, the CNI will notify the LSM of 
the returned data via a callback defined in the LSM document. 

CNIAcquireAccessToken 
eStreamStatus CNIAcquireAccessToken( 

IN uintl28 SubscriptionID, 

IN string Username, 

IN string Password 

); 

CNIAcquireAccessToken will cause the CNI to contact a SLiM server to retrieve an ac- 
cess token. The CNI is responsible for issuing retries if no response is received for a re- 
quest. The CNI will call the appropriate LSM callback function when the data come 
back. 

CNIRenewAccessToken 
eStreamStatus CNIRenewAccessToken( 

IN AccessToken Token, 

IN string Username, 

IN string Password 

); 

CNIRenewAccessToken will enqueue a request for access token renewal. When the re- 
sponse comes back, the CNI will dispatch the returned data to the appropriate LSM call- 
back routine. 
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CNIReleaseAccessToken 
eStreamStatus CNIReleaseAccessToken( 

IN AccessToken Token, 

IN string Username, 

IN string Password 

); 

CNIReleaseAccessToken will enqueue a request for releasing an access token When 
the response conies back, the CNI will dispatch the returned data to the appropriate LSM 
callback routine. 

CNIRefreshAppServerSet 
eStreamStatus CNIRefreshAppServer( 

IN AccessToken Token, 

m uinm BadQOS, 

IN uint32 NoService 

); 

CNIRefreshAppServerSet will enqueue a request for refreshing the app server set 
When the response comes back, the CNI will dispatch the returned data to the appropriate 
LSM callback routine. 

The client networking component will also have routines for getting and setting tunable 
parameters. " 

CNISetParameter 
eStreamStatus CNISetParameter( 

IN NWTunableParameter type, 

IN void *value 

); 

CNISetParameter sets a parameter. The actual type of value is determined by type. 

CNIGetParameter 
eStreamStatus CNIGetParameter( 

IN NWTunableParameter type, 

OUT void *value 

); 

CNIGetParameter queries the current value of a parameter. The actual type of value is 
determined by type. 

Component design 

J hC i " t ^ r " al i or g anization °f client networking depends on the mechanisms available 
from EMS. Internally, the CNI interface functions put requests on a queue and one or 
more threads services these requests by using the EMS to send messages to servers 
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Synchronous Server Calls 



If EMS only provides a synchronous messaging service, a single thread will be used to 
perform all necessary actions. The CNI interfaces will put appropriate requests on the 
network request queue. They will also wake up the network communication thread if 
necessary. 

The network communication thread's job is relatively simple. When it wakes up it per- 
forms the following tasks: 

- choose a set of requests to be coalesced and remove these from the request queue 

- retrieve a server set via LSMGetAppServerSet or LSMGetSLiMServerSet, and choose a 
particular server for this request 

- make a synchronous EMS call to send the request 

- dispatch the response to the appropriate LSM or ECM callback 

If the synchronous messaging mechanism becomes a performance bottleneck, we can 
have multiple network communication threads to increase concurrency. 




Asynchronous Server Calls 

The asynchronous case is a little bit more complex. Because of the proposed asynchro- 
nous call architecture, the client NW requires three threads. The CNI interfaces work just 
as they do in the synchronous case. They put requests on the network request queue and 
wake up the network send thread. However, the actions performed by the CNI's worker 
threads differ in the asynchronous model. 
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The network send thread is periodically awoken, and i t coalesces requests off the NW 
request queue and sends them to the server. Unlike in the synchronous model, this thread 
does not synchronously wait for the request to come back from the server. Instead, it 
simply sends requests until the queue is empty, then goes back to sleep. 

The network receive thread waits for responses to come back from any server. Because 
of the EMS's asynchronous call implementation details, this thread posts returned data to 
a queue of responses to be handled by another thread. The network receive thread is also 
responsible for handling timeouts and reissuing those network requests on different serv- 
ers. 

Finally, the response dispatch thread pulls responses off the response queue, and handles 
the work of dispatching them appropriately. 
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Handling Network Failure 

When the client networking component is notified of a message failure by the EMS, the 
client worker thread will attempt to reissue the request on a different server. 

Coalescing Multiple Requests 

The CNI will coalesce multiple page requests that come from the LSM into a single re- 
quest to an application server. Multiple pages requests for the same application may be 
coalesced. No other types of requests may be coalesced, including page requests for dif- 
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ferent applcations. The CNI will not produce requests larger than the maximum allowed 
by the application server. 

Handling Persistent Failures 

There will be some persistent failures that will result in the network being unable to ful- 
fill page requests in a timely fashion. This may be due to network or server failure. 
(These may be indistinguishable from the CNI's point of view.) When the CNI has failed 
to satisfy a request for a certain amount of time, it will need to ask the user if he wants it 
to continue retrying, or if it should let the application terminate. It will do this via the 
CUIAskUserYesNo() interface. The client software control panel should include an op- 
tion to always wait until the server is available, and never ask the user if he wants the ap- 
plication terminated. 

Testing design 
Unit testing plans 

The testing harness for the networking component will be a set of dummy EMS drivers 
and a dummy NW client. The dummy EMS driver will be capable of performing a vari- 
ety of actions, including returning appropriate responses, returning inappropriate re- 
sponses, and timing out without any response. The dummy NW client will have knowl- 
edge about the expected EMS behavior, and will verify that the data it gets back from the 
network component are as expected. 

Stress testing plans 
Failure testing plans 

The client NW is the sole component responsible for implementing server failover. In 
order to test this code, it is necessary to implement a server with predefined bad behavior. 
The server failure modes that must be tested include 

- server that accepts a connection on a socket but doesn't respond to any requests 

- server that closes the socket before sending a response 

- server that closes the socket in the middle of a response 

- server that sends a partial response and then just stops 

- server that satisfies n requests then closes the socket or refuses to service more 

It is important that we cover scenarios that look like network failures and ones that look 
like server failures. (Are there other failure modes that are interesting?) 

Cross-component testing plans 

Cross-component testing of the client NW includes integration testing with the EMS, the 
LSM, and the prefetcher. Testing with the EMS can be performed in a manner similar to 
unit testing in conjunction with a specially written server. Testing with the LSM or pre- 
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fetcher can be performed in isolation by writing drivers for either the LSM or prefetcher, 
and using a dummy or real EMS. I'm not sure if this sort of testing is worth the effort to' 
write the appropriate harnesses. Verifying the output of such a combined system is cer- 
tainly trickier than testing any component in isolation. 



Open Issues 
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Functionality 

This document represents the low level design of the Client eStream User Interface 
[CUI], an eStream client module that: 

• supports asynchronous communication from various eStream client components 
to the user, optionally soliciting responses 

• provides a visual indication to the user that the eStream client software is running, 
via the presence of the eStream systray icon 

• displays ABOUT and HELP information to the eStream client user, including a 
link to a web site with FAQs and access to support with respect to eStream 

• interfaces with a (planned but not yet designed) client ASP user interface, which 
allows the user to view ASP billing information, view/change ASP account in- 
formation, subscribe/unsubscribe applications, receive ASP-specific ABOUT and 
HELP information, etc, from the client without requiring that the user manually 
point his browser at the ASP website and login 

• allows an eStream user to request a variety of [optional/advanced] eStream client 
management functions (described below) 

Please note that the eStream CUI is not used to launch eStream applications; eStream ap- 
plications are launched (like locally-installed applications) via Start/Programs or via 
desktop icons. The eStream CUI is considered to be a utility interface (in Windows UI 
parlance) and will follow the Windows UI conventions of this category of program. 

Data definitions 

eStream client software will be developed in a manner which facilitates porting its user 
messages and interfaces to languages other than English. Since the CUI is responsible 
for displaying messages and help text to client users, it is the natural module to "own" the 
issue of localization. 

Windows Resource Files will be used for all user messages; CUI's callers will pass han- 
dles to messages they wish displayed. All user displays/menus/bitmaps will be devel- 
oped in a manner consistent with the guidelines outlined in "Developing Internationa] 
Software for Windows 95 and Windows NT" by Nadine Kano [MSDN Online Library]. 
Please note that eStream 1 .0 is targeted to the English language market, and translation of 
the Resource Files to other languages is not planned to be completed prior to first product 
shipment. 

The AppName, VersionName, & Message info supplied in the eStreamAppInfo are in the 
same language as the version of the app (i.e., Japanese Word => Japanese). 
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Interface definitions 

Client Component Acronyms 

• CES: Client EStream Startup 

• CNI: Client Network Interface 

• CUI: Client User Interface 

• ECM: EStream Cache Manager 

• EPF: EStream PreFetch/Fetch - 

• LSM: License Subscription Manager 

From <various client components> to CUI: 

• CUIInformUser(IN Message, IN OptionalHelpInfo) 

• CUIAskUserYesNo(IN Message, IN OptionalHelpInfo, IN CheckAskAgain, 
OUT Response, OUT DontAskAgain) 

• CUIAskUserPassword(IN Message, OUT Response, OUT Retain) 

From CUI to CES: 

• CESSetActivate(IN Boolean ActivateAtLogin) 

• CESDeactivate(VOID) 

From CUI to LSM: 

• LSMGetAspList(OUT NumAsps, OUT AspID[]) 

• LSMGetAspInfo(IN AspID, OUT ASPWebServerName, OUT UserName) 

• LSMUpdateAHSubscriptionStatus(VOID) 

• LSMGetAppList(IN AspID, OUT NumApps, OUT AppTD[]) 

• LSMGetAppInfo(IN AppID, OUT AppName, OUT VersionName, OUT Mes- 
sage, OUT AppInstallStatus, OUT AppUpgradeStatus) 

• LSMInstall(IN AppID, OUT InstallStatus) 

• LSMUninstall(IN AppID, OUT UninstallStatus) 

• LSMUpgrade(IN AppID, OUT UpgradeStatus) 

From CUI to ECM: 

• ECMSetCachelnfo - causes ECM to recheck registry for cache size 

• ECMGetCachelnfo 

From CUI to EPF: 

• EPFPrefetchSet(IN OnOff) 
From CUI to CNI: 

• CNIGetParameter(IN ParmName, OUT Parm Value) 

• CNISetParameter(IN ParmName, IN ParmValue) 
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Component design 

Asynchronous Messages to eStream User 

There are several (hopefully rare) situations that arise asynchronously, about which com- 
ponents of the eStream client software inform the user. In some cases, these situations 
necessitate user response. CUI delivers asynchronous messages via pop-up dialog boxes. 
If no user input is needed, other than confirmation that the user has seen the message, the 
box displays text with a simple Continue button [CUIInformUser]. If the message is a 
question to which a yes/no response from the user is sought, the pop-up dialog box con- 
tains Yes and No buttons with the recommended selection emphasized [CUIAskUserY- 
esNo]. If the message requires that the user enter a password, the pop-up contains an edit 
box in which character echoing is shielded [CUIAskUserPassword]. Help buttons with 
supporting amplifying passages are supplied in situations in which extra information can 
aid user understanding. 

eStream Systray Icon 

EStream client software can be activated automatically at user login, manually via a 
Start/Program entry, and automatically when an EFSD request arrives and EFSD is inac- 
tive. [Please see the CES design materials for more information on activation.] The 
eStream client software must be active for the user to execute eStream applications; the 
presence of the systray icon indicates that the software is active. Right-clicking the icon 
engenders a pop-up menu from which the user can reach about/help info, ASP client ac- 
count interfaces, and eStream client management functions. 

EStream About/Help Information 

The chief external interface involved in eStream-specific ABOUT+HELP is launching a 
browser and passing arguments to it. The user's default browser is used for this. Please 
note that the user's default browser must be one of the browsers specified as required for 
eStream 1 .0 (IE4 or higher, Navigator 4 or higher). 

ASP Client Account Interfaces 

It is beyond the scope of this document to describe the interfaces with a (planned but not 
yet designed) client ASP user interface, which allows the user to view ASP billing infor- 
mation, view/change ASP account information, subscribe/unsubscribe applications, re- 
ceive ASP-specific ABOUT+HELP information, etc., from the client without requiring 
that the user manually point his browser at the ASP website and login. However, the CUI 
implementation includes bringing up a browser and passing arguments to it, which is key 
to supporting the success of this future functionality. 
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eStream 1 .0 Client User Interface Low Level Design 



EStream Client Management Functions 

The typical user should not normally need to use the eStream client management func- 
tions available from the CUI. Subscription information is synchronized automatically, 
applications are installed/uninstalled when the subscription information is synched, 
eStream client software is activated at login & deactivated at logout, the cache is right- 
sized, etc. The functions included in this section are supplied for use in special circum- 
stances and/or in support situations. 

eStream client management functions include the following: 

• For ASP accounts known to this client: 

o View account information [LSMGetAspList, LSMGetAspInfo] 

o Request synchronization of application subscription information [handled 

automatically at client eStream activation, included here for flexibility] 

[LSMUpdateAlISubscriptionStatus] 

• For application subscriptions known to this client: 

o View subscription information [this may engender a synch] [LSMGetAp- 
pList, LSMGetAppInfo] 

o Install subscribed application [handled automatically at synch time, in- 
cluded here for flexibility] [LSMInstall] 

o Uninstall application [handled automatically at synch time, included here 
for flexibility] [LSMUninstall] 

o Upgrade application to latest version [handled automatically, flexibility] 
[LSMUpgrade] 

• For eStream client software: 

o Specify automatic activation at login vs manual activation via pro- 
gram/start [it is proposed that we assume the former is always the case] 
[CESSetActivate] 

o Any general user preference wrt terminating app for very slow network re- 
sponse (always wait vs. terminate if too slow) 

• For eStream client cache [primarily for users who do not allow eStream to right- 
size their cache]: 

o Display current size and utilization [obtained from registry, ECMGet- 
Cachelnfo] 

o Set size: allow eStream to right-size vs. manually increase/decrease size 

[ECMSetCachelnfo] 
o Disable/enable prefetching [EPFPrefetchSet] 

• For eStream client network interface: 

o Display recent eStream effective bandwidth [possibly displayed as hover- 
over on eStream systray icon] [CNIGetParameter] 

o View/Set proxy IP address [may use info from control panel internet op- 
tions widget] [CNIGetParameter, CNISetParameter] 

o View/Set connection time-out value [CNIGet/SetParameter] 

o View/Set number retries value [CNIGet/SetParameter] 

• Deactivate eStream [CESDeactivate] 
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eStream 1 .0 Client User Interface Low Level Design 

Testing design 
Unit/Coverage testing plans 

For the eStream client management functions and the ABOUT+HELP displays, a UI 
scripting tool like Rational^ Visual Test will be used to automate testing. Unit testing 
will involve creating driver keystrokes that exercise all menu selections and stubs of 
other client modules that will report as many asynchronous error conditions as possible. 
Using Rationale PureCov tool, PFA coverage should be as close to 100% as possible. 

Cross-component testing plans 

With respect to the eStream client management functions, the CUI has many interfaces 
with LSM, and hooking the two together is a win-win in terms of facilitating the testing 
of both; this combination will be the first cross-component integration. CUTs unit test 
scripts are expected to continue to be useful in driving the testing of the CUI/LSM com- 
bination. Other client management interfaces will be added and tested as the associated 
components are completed, with continued emphasis on ensuring that asynchronous error 
messages are provoked. 

Stress testing plans 

Need/strategy unclear. 

Upgrading/Supportability/Deployment design 

A chief motivation for many of the user interfaces in the CUI design, particularly in the 
eStream client management functions area, is user support and deployment support. 

Implementation Issues 

• Investigate obtaining ProxyIP Address from control panel internet options widget. 
[BTW, why aren't our competitors doing this?] 

• Some items accessed by CUI are per-user (ASP, applications, activation) and 
some items are per-client (cache size, network interface). We need to make sure 
this is clearly communicated in the UI. 

• API needs to be available so that every button/widget can be manipulated pro- 
grammatically to facilitate testing. 
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eStream 1.0 Client eStream User Interface Straw Man 



Anne Holler 




* Version 1.3 



Introduction 

This document presents background information related to the Client eStream User 
Interface [CUI], an eStream client module that: 

• supports asynchronous communication from various eStream client components 
to the user, optionally soliciting responses 

• provides a visual indication to the user that the eStream client software is running, 
via the presence of the eStream systray icon 

• displays ABOUT and HELP information to the eStream client user, including a 
link to a web site with FAQs and access to support with respect to eStream 

• interfaces with a (planned but not yet designed) client ASP user interface, which 
would allow the user to view ASP billing information, view/change ASP account 
information, subscribe/unsubscribe applications, receive ASP-specific ABOUT 
and HELP information, etc, from the client without requiring that the user 
manually point his browser at the ASP website and login * 

• allows an eStream user to request a variety of [optional/advanced] eStream client 
management functions (described below) 

The document is intended to provide a baseline for discussions prior to proceeding with a 
detailed low-level design. 

Please note that the eStream CUI is not used to launch eStream applications; eStream 
applications are launched (like locally-installed applications) via Start/Programs or via 
desktop icons. The eStream CUI is considered to be a utility interface (in Windows UI 
parlance) and will follow the Windows UI conventions of this category of program. 

Asynchronous Messages to eStream User 

There are several (hopefully rare) situations that arise asynchronously, about which 
components of the eStream client software will inform the user. In some cases, these 
situations necessitate some user response. CUI delivers asynchronous messages via pop- 
up dialog boxes. If no user input is needed, other than confirmation that the user has seen 
the message, the box will display the text along with a simple Continue button 
[CUIInformUser]. If the message 1 is a question to which a yes/no response from the user 
is sought, the pop-up dialog box will contain Yes and No buttons with the recommended 
selection emphasized [CUIAskUserYesNo]. If the message requires that the user enter a 
password, the pop-up will contain an edit box in which character echoing is shielded 
[CUIAskUserPassword]. Help buttons with supporting amplifying passages are 
supplied in situations in which extra information can aid user understanding. 
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Please note that the eStream client software is expected to be developed in a manner 
which facilitates porting its user messages and interfaces to languages other than English. 
Hence, messages are to be obtained from a catalog or resource file to ease translation. 



eStream Systray Icon 

The eStream client software must be active for the user to execute eStream applications; 
the presence of the systray icon indicates that it is active. Right-clicking the icon 
engenders a pop-up menu from which the user can reach help/about info, ASP client 
account interfaces, and eStream client management functions. 

EStream About/Help Information 

The chief external interface involved in eStream-specific ABOUT+HELP is launching a 
browser and passing arguments to it. It is expected that the browser utilized for this 
feature will be Internet Explorer; if so, we will require that Internet Explorer not be 
uninstalled from the user's windows platform (which is difficult to do, as demonstrated at 
Microsoft's antitrust trial). Please note that the user can still employNetscape Navigator 
to connect to the ASP, as the eStream 1 .0 requirements document specifies; this is a 
separate issue of what browser is used for the client pass-through to ASP account 
functionality. 



ASP Client Account Interfaces 

It is beyond the scope of this document to describe the interfaces with a (planned but not 
yet designed) client ASP user interface, which weill allow the user to view ASP billing 
information, view/change ASP account information, subscribe/unsubscribe applications, 
receive ASP-specific ABOUT+HELP information, etc, from the client without requiring 
that the user manually point his browser at the ASP website and login. However, the CUI 
implementation includes bringing up a browser and passing arguments to it, which is key 
to supporting the success of this future functionality. 



EStream Client Management Functions 

The typical user should not normally need to use the eStream client management 
functions available from the CUI. Subscription information is synchronized 
automatically, applications are installed and uninstalled when the subscription 
information is synched, eStream client software is activated at login, the cache is right- 
sized, etc. The functions included in this section are supplied for special circumstances 
and/or support situations. 
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eStream client management functions include the following: . 

• For ASP accounts known to this client: 

o View account information [LSMGetAspList, LSMGetAspInfo] 

o Request synchronization of application subscription information [handled 

automatically at client eStream activation, included here for flexibility] 

[LSMUpdateAllSubscriptionStatus] 

• For application subscriptions known to this client: 

o View subscription information [this may engender a synch] 

[LSMGetAppList, LSMGetAppInfo] 
o Install subscribed application [handled automatically at synch time, 

included here for flexibility] [LSMInstall] 
o Uninstall application [handled automatically at synch time, included here 

for flexibility] [LSMUninstall] 
o Upgrade application to latest version [handled automatically, flexibility] 

[LSMUpgrade] 

• For eStream client software: 

o Specify automatic activation at login vs manual activation via 

program/start [it is proposed that we assume the former is always the case] 
[CESSetStartup] 

• For eStream client cache [primarily for users who do not allow eStream to right- 
size their cache]: 

o Display current size and utilization [obtained from registry] 

o Set size: allow eStream to right-size vs. manually increase/decrease size 

[ECMSetCacheSize] 
o Disable/enable prefetching [EPFPrefetchSet] 

• For eStream client network interface: 

o Display recent eStream effective bandwidth [possibly displayed as hover- 
over on eStream systray icon] [CNIGetEffectiveBandwidth] 

o View/Set proxy IP address [may use info from control panel internet 
options widget] [CNIGetProxylPAddress, CNISetProxylPAddress] 

• Deactivate eStream [CESShutdown] 

CUI Interfaces To/From Client Components 

Client Component Acronyms 

CES: Client EStream Startup 
CNI: Client Network Interface 
CUI: Client User Interface 
ECM: EStream Cache Manager 
EPF: EStream PreFetch 
LSM: License Subscription Manager 

From CUI to CES: 
CESSetStartup(IN Boolean ActivateAtLogin) 
CESShutdown(VOID) 
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From CM to LSM: 

LSMGet AspLi st(OUT NumAsps, OUT AspID[]) 

LSMGetAspInfo(IN AsplD, OUT ASPWebServerName, OUT UserName) 
LSMUpdateAllSubscriptionStatus(VOID) 
LSMGetAppList(IN AsplD, OUT NumApps, OUT AppID[]) 
LSMGetAppInfo(IN AppID, OUT AppName, OUT VersionName, OUT Message, 

OUT AppInstallStatus, OUT AppUpgradeStatus) 
LSMInstall(IN AppID, OUT InstallStatus) 
LSMUninslall(IN AppID, OUT DeinstallStatus) 
LSMUpgrade(IN AppID, OUT UpgradeStatus) 

From CUI to ECM: 

ECMSetCacheSize(VODD) - causes ECM to recheck registry for updated cache size 

From CUI to EPF: 
EPFPrefetchSet(IN OnOff) 

From CUI to CNI: 

CNIGetProxyIPAddress(OUT ProxylPAddress) 
CNISetProxyff Address(IN ProxyIP Address) 

CNIGetEffectiveBandwidth(OUTEstreamRecentEffectiveBandwidth) 

From <various client components> to CUI: 
CUIInformUser(IN Message, IN OptionalHelpInfo) 
CUIAskUserYesNo(IN Message, IN OptionalHelpInfo, IN CheckAskAgain, 

OUT Response, OUT DontAskAgain) 
CUIAskUserPasswordflN Message, OUT Response, OUT Retain) 

Issues 

• Resolve general issue of whether eStream client management functions should be 
included in product at all (opinions on both sides). 

• Within client management functions, should prefetch enable/disable be included? 

• Should CNI get ProxylPAddress from control panel internet options widget and 
should we therefore remove this menu item & interface from CUI? 

• Will cache utilization be posted to the registry or available somehow? 

• What is the interface to obtain recent effective eStream network bandwidth? 
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Functionality 

The eStream cache manager implements much of the client-side functionality for 
handling the eStream file system. The cache manager handles all file system requests 
made by the operating system by reading information from the cache or by passing the 
requests along to the profiling and prefetching component to fetch missing data from the 
network. 

The cache manager will initially be implemented in user space, but it may be useful to 
migrate it to the kernel for improved performance. In user space, it will be part of the 
eStream client process. In the kernel, it will probably be a device driver distinct from the 
eStream file system driver. 

The cache manager manages the on-disk cache of file system data, and the in-memory 
data structures for managing this cache. It does not manage prefetching of data from the 
server; that is the role of the eStream Profiling and Fetching (EPF) component. A 
separate networking component handles the network traffic. This component will also be 
described separately. 

Since there is no overall discussion of the client architecture at a more detailed level than 
the high level design, this document will cover that as well. 

Multiple cache page files will be supported. Each cache page file may be up to 2 GB in 
size. Different cache files may reside on different or the same logical disk (i.e. Windows 
drive letter.) 

Data type definitions 

An application ID uniquely identifies an eStream application. lust what constitutes "one" 
eStream application is not entirely defined, but different "builds" of the "same" app will 
be considered different eStream applications. For example, the Chinese-language version 
of Office is a different eStream application than the English-language version. 

typedef uintl28 ApplicationID; 

The eStream page number is the data type used to describe a page number within a 
particular file. Note that this is a page offset, not a byte offset. For eStream 1 .0, the 
cache manager will only support 2 GB cache files. 

typedef uint32 EStreamPageNumber ; 
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The fileld is used to uniquely identify a file within the universe of all eStream files across 
all eStream applications. 

typedef struct { 

ApplicationID App, 

int32 File 
} fileld; 

The eStream page size is the fundamental size for eStream requests. This size is in bytes, 
tfdefine ESTREAM_PAGE_SIZE 4 096 

The eStream file system uses the file time format of the Windows operating system. If 
the client runs on a system with a different native time format, the client software will be 
responsible for translating between the native format and the eStream format. The 
Windows data format is a 64-bit counter of the number of 100-nanosecond periods since 
January 1, 1601. 

EStream metadata is the file information supported by the eStream file system. This 
metadata is independent of the client or server operating system. 

typedef struct 
{ 

uint64 CreationTime; 
uint64 AccessTime; 
uint32 FileSize; 
uint32 FileSystemAttributes; 
uint32 EStreamAttributes; 
} Metadata; 

The eStream inode contains the layout of a file in the cache. Each inode has the 
following structure: 

typedef struct 

^ Fileld Id; /* ID of this file; search parent for 

name*/ 

Metadata Metadata; 

FilelD Parent; /* parent directory's file id */ 
uint32 NumPages; 
Pagelnfo * Pages; 
} EStreamlnode; 

The Pagelnfo array is variable sized. There is one entry in the pages array for each page 
in the file (not for each page cached, since we need to know whether the pages are 
present or not...) Note that the inode is only used in the "robust" implementation. 
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typedef struct 

EStreamPageNumber CachePageNumber ; 
PageStatus Status; 
unsigned char Priority; 
PageChecksum Checksum; 
} Pagelnfo; 

The page number doesn't require the 32 bits, since pages are 4096 bytes long. The extra 
bits will be used to encode which cache file this page resides in. The priority field is a 
number representing this page's priority for being kicked out of the cache. How exactly 
this field is used hasn't yet been determined. The checksum is a (fast) page checksum 
that can be used to validate the contents of this page. Note that it will be useful to have a 
slower, more effective checksum for development and a faster (but less thorough) 
checksum for deployment. 

The page status is an enumeration for the page's locking status (these are described in 
more detail later: 

typedef enum 

{ 

PS_INVALID, 
PS_CLEAN_UNLOCKED , 
PS_CLEAN_LOCKED , 
PS_DIRTY_UNLOCKED , 
PS_DIRTY_LOCKED , 
PS_IN_FLIGHT 
} PageStatus; 

Note that this describes the layout of the tables in memory; how these data structures are 
represented on disk is described later. 

The EFSD file handle is a small integer passed between the EFSD and the ECM. This is 
used opaquely by the EFSD and is used as an index into an open file table by the ECM. 

typedef uint32 EFSDFileHandle; 

The ECM request type specifies the request type to the rest of the system. Note that some 
"requests" are used to inform the prefetcher about the events handled solely by the ECM, 
and do not actually request that any particular action be taken by the prefetcher. 

typedef enum 
{ 

ERT_READ, 
ERT_WRITE , 
ERT READ HIT, 
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ERT_WRITE_HIT 
} ECMRequestType ; 

The ECM request is a request descriptor that is used in various lists within the cache 
manager. These lists are doubly-linked, circular lists. 

typedef struct _ECMRequest 

uint32 RequestID; /* same as EFSD request id */ 
ECMRequestType RequestType; 

union {} Parameters; /* union of all parameters*/ 
struct _ECMRequest *next; 
struct _ECMRequest *prev; 
} ECMRequest; 

The cache manager must maintain an array of files that have currently been opened by 
the EFSD. This array will be statically allocated. This will put a limit on the number of 
files that may be opened concurrently on the eStream file system. The elements of the 
array are the following: 

typedef struct 
{ 

uint32 Valid ; 
fileld File; 

HANDLE OpenFile; /* for simple implementation */ 
eStreamlnode *Inode; /* for robust implementation */ 
} OpenFilelnfo; 

The cache manager maintains a hash table containing information about each application 
that currently has open files. The hash table is indexed by app ID, and contains the 
following active app information records: 

typedef struct 

^ AppID App; /* identity of this app */ 

uint32 OpenFiles; /* # of open files */ 

uint32 HaveAccessToken; /* boolean */ 
} ActiveAppInfo; 

The ECM will use this table to quickly determine whether it should continue processing a 
request it gets from the EFSD, or if the request should be passed to the LSM to ensure 
that an access token is available. See the section below on ECM-LSM interaction for 
more details. 

The LSM uses the access token state to specify a state for an access token. Right now, 
we only plan to support valid and invalid, but it may be interesting in the future to allow 
already opened files to be read, but no new files to be opened. 
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typedef enum 
{ 

ATS_INVALID, 
ATS_VALID, 
ATS_VALIDJNJO_OPEN 
} AppTokenState; 



Interface definitions 

The ECM exports the following interfaces for operating on the cache. They may be 
called by the cache manager, prefetches or networking component. (Not all components 
are expected to call all interfaces; see each interface description for more details.) 

Note that the cache interfaces are defined at a very high level as the actions that may be 
performed on the cache by the components, such as enqueuing a new request. They have 
been defined this way so that these intrinsic operations can be implemented correctly 
once and limit the possibility that an individual component will not perform proper 
actions. 

ECMReservePage 
eStreamStatus ECMReservePage( 
IN fileld File, 

IN EStreamPageNumber Page, 
IN ECMRequest * Request 

); 

ECMReservePage reserves a page in the cache for a request. This interface is called by 
the prefetching component, and will send a request to the network component. Logically, 
this interface reserves an empty cache page for this request (if one is available), puts this 
request on the "in flight" queue, and calls on the network to request the page (unless it is 
already in flight.) 

ECMIsPagelnCache 

eStreamStatus ECMIsPageInCache( 
IN fileld File, 

IN EStreamPageNumber Page 

); 

ECMIsPagelnCache returns TRUE if the specified block is in the cache, and FALSE 
otherwise. It is used by the EPF to determine if it should prefetch a block; normally, the 
EPF would choose not to prefetch something that is already in the cache. Note that it 
would be a good idea for the prefetcher to adjust the priority of a page that it thinks it 
wants to prefetch, so that they are less likely to be evicted from the cache before they are 
needed. 

ECMDeplanePage 
eStreamStatus ECMDeplanePage( 
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IN fileld File, 

IN EStreamPageNumber Page, 

IN ch ar Buffer [ESTREAM PA GESIZE] 

); 

ECMDeplanePage performs all the necessary actions for writing a page coming off the 
network into the cache and back to the EFSD. This consists of copying the page into the 
cache, remove all pending requests for this page from the in flight list, marking the page 
as clean/unlocked, and returning the page to the EFSD for each in flight request. 

ECMReadPage 
eStreamStatus ECMReadPage( 
IN fileld File, 

IN EStreamPageNumber Page, 
IN ECMRequest *Request 

); 

ECMReadPage performs all the necessary actions for attempting a page read from the 
cache. The cache is checked to see if it contains the page; if so, the page is copied to the 
buffer, the EPF is notified of the hit, and appropriate status is returned. Otherwise, this 
page is put on the queue for requests pending to the prefetching component, and 
appropriate status is returned. 

ECMWritePage 
eStreamStatus ECMWritePage( 
IN fileld File, 

IN EStreamPageNumber Page, 
IN ECMRequest *Request 

); • - u 

ECMWritePage performs all the necessary actions for attempting to wnte a page in the 

cache. Note that this could be somewhat more complex than a read, because a partial 

write to a page might necessitate reading the page from the server before writing the 

partial page to the cache. 

The following interfaces are the abstract interfaces that the ECM will use to communicate 
with the EFSD. Hiding the EFSD's raw DeviceloControls behind these interfaces will 
help make porting the ECM into the kernel easier, should we decide to do that. 

ECMSetTokenState 
eStreamStatus ECMSetTokenState( 

IN Appld App, 

IN AppTokenState State 

); 

ECMSetTokenState is called by the LSM to indicate to the ECM that a token has 
become available or has expired. The main effect of this interface is to update the state of 
the specified application in the active app table. See the ECM-LSM interaction below for 
more details. 
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ECMGetCachelnfo 
eStreamStatus ECMGetCacheInfo( 

OUT UNICODE_STRING Location, 

OUT uint32 *CurrentSize, 

OUT uint32 *MaximumSize 

ECMGetCachelnfo is called by the client user interface to find out where the ECM 
cache is located and its current and maximum size. Location is an absolute path name of 
the cache file. 

ECMSetCactaelnfo 
eStreamStatus ECMSetCacheInfo( 

IN UNICODE_STRING Location, 

IN uint32 MaximumSize 

ECMSetCachelnfo is called by the user interface when a new cache location or size has 
been requested. Note that the cache manager may only begin using the new cache 
information after a restart of the client software (which may only occur on client machine 
reboot.) The client UI will call this interface when it wants to make a change; the ECM 
is responsible for actually resizing the cache and making any changes necessary to 
persistent storage (i.e. the registry). 

EFSDGetRequest 
eStreamStatus EFSDGetRequest( 

OUT EStreamRequest **Request 

EFSDGetRequest reads the next request from the EFSD, including any parameters that 
need to be passed. This may involve one or more DeviceloControl calls to the EFSD. 
EFSDGetNextRequest is responsible for allocating memory for this request, and an 
EFSDCompleteRequest call will be repsonsible for deallocating the memory. 

EFSDCompleteRequest 
eStreamStatus EFSDCompIeteRequ est( 

IN EStreamRequest *Request, 

IN ECMErrorCode Status 

EFSDCompleteRequest will be called for each request that is received by the ECM via 
EFSDGetRequest. status indicates the completion status for this request, and may 
indicate success, a retry, or a particular failure condition. Non-persistent errors will be 
handled by the ECM internally or by requesting a retry of a particular request. Errors 
reported to the EFSD will be propagated up the file system stack. 



Overall Client Architecture 



Omnishift Confidential 



Page 7 



The eStream client will have various types of threads in order to perform its work. The 
basic architecture is illustrated by the following diagram. 



Get access token, release access token 
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The FSD worker thread will pull requests from the FSD. It will return data for requests 
that can be satisfied immediately. Any request that requires information that is not 
currently in the cache will be put on a queue for the prefetching thread to handle. 

The profiler will receive all cache misses from the FSD worker thread. Using its own 
data structures (which may include information about recent cache misses in addition to 
information about general prefetch patterns), it will decide which blocks it should 
prefetch. Demand fetch and prefetch requests are sent to the network component. The 
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only way demand fetches and prefetches are treated differently by the network 
component is that demand fetches are sent to the EFSD while prefetches are not. 

The network thread will manage open connections to app servers and retry requests that 
time out When data comes back from the network, the network thread will copy the 
returned buffer into the cache and to the FSD, if the request was a demand miss. 

The cache manager consists of the EFSD worker thread and the APIs to access the cache 
index, the data blocks, and various queues used by threads in the client. 

Not shown on the diagram is an error thread. This thread is responsible for calling the 
client UI module indicating appropriate error messages and waiting for the user's input. 
When any component decides that it has an error condition that requires user input, it 
calls ECMReportError with the request and an appropriate error condition, which will 
be enqueued for the error thread to handle. For example, when the network interface 
times out reading a page from an application server enough times, it will call 
ECMReportError. When the error thread gets to this request in the queue, it will ask 
the user if he wants to wait until the app server is available or allow the application to 
terminate. 

ECM-LSM Interaction , _„ . , 

The ECM-LSM interaction is a relatively simple one. The LSM notifies the ECM when 
it first receives an access token and when its access token expires. It does this via the 
ECMSetTokenState interface. The ECM keeps track of each application that has had 
files open, and whether or not we have an access token for each of these apps. 



App ED 


# of open files 


Have access token? 















Note that the LSM need not notify the ECM of mundane events like renewals as long as 
some token is valid. Also, the ECM does not keep track of the token itself, just whether 
or not we have a valid one. An additional nicety of this approach is that we could allow 
the ECM to satisfy requests out of the cache as if we have an access token, without 
actually having one. 

When it receives a request, the ECM checks its table to determine if an access token is 
available. If it is, it handles the request as normal. If not, it asks the LSM to acquire an 
access token via LSMGetAccessToken. The LSM may return that it has a token, in 
which case the ECM will continue to process the request, or the LSM may say it doesn't 
have a token, in which case the LSM takes ownership of the request and will reissue the 
request when the access token is available. 
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When the number of open files drops from 1 to 0, the ECM will mark the token as invalid 
in its table and call LSMReleaseToken. The LSM may choose not to renew access 
tokens that have been released. 

Component design 

Two cache organizations will be presented. One is suitable for a quick implementation 
but doesn't lend itself particularly well to high performance or easy manageability; the 
other will be more difficult to implement but should provide better performance. I will 
first describe some data structures that are shared by both designs, then go into the 
specifics of each design. 

Common Data Structures and Algorithms 

Certain request lists are common to both cache organizations. One is a queue between 
the FSD worker thread and the prefetching thread for demand fetches that have not yet 
been seen by the prefetcher. The other is a list of all requests for pages that are "in 
flight." Requests from the in flight list are removed when they have been satisfied. The 
in flight list is unsorted and searched whenever a request comes back for requests that 
match the returned page. If the performance of this data structure becomes an issue, we 
will change its organization for faster lookup. 

Both request lists use the request data structure described above. 

The ECM will maintain an array of files currently opened by the EFSD. On file opens, 
an empty location in this table will be allocated for the newly opened file, and the index 
to that entry returned as the file handle. (Note that the way the interface between the 
ECM and the EFSD is defined, it is an error to open an already opened file. The cache 
manager will have to detect such cases and report an error, but it will not keep a reference 
count of the number of opens on each file.) This mechanism will allow the ECM to keep 
track of the volumes that currently have opened files as well as abstracting the 
client/server file ids away from the kernel driver. (This might allow us to update the 
client/server protocol without rewriting the EFSD.) 

Easier Implementation 

The cache will be implemented as a directory tree on the user's hard drive that parallels 
the eStream file system. Each file will contain a header and an array of status bytes in 
addition to the data blocks that the file contains. The array of status bytes has one byte 
for each page in the file. Each byte indicates the current status of that page in the file. 
(Pages have several different states, so a simple bit per page is not sufficient.) Each file 
will thus look like 



Header 



Page Status Bytes 



File contents page 0 
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File contents page 1 



The header is defined as: 
typedef struct 

{ 

uint32 magicCookie; 

uint32 headerLength; /* Length of this header, in bytes */ 

fileld fileid; /* for sanity checking */ 

uint32 length; /* Length of the file, in bytes */ 

uint32 firstPage; /* Offset to the first page in the file 

*/ 

Metadata metadata; 
} ECMCacheFileHeader ; 

The page status bytes begin immediately following the header, and this area is padded 
with zeros to a page boundary. The first page of the file's contents (and thus each 
following page of file contents) will therefore begin on a page boundary. 

Note that one issue with this design is that files that approach the file size limit of the 
underlying file system cannot be represented, due to the overhead with the header and 
bitmap. If this design is used solely for early engineering efforts, then this limitation is 
acceptable. If we have to work around this limitation, one way to do it is to make the 
headers and page status bytes reside in a separate file or files. 

Directory contents would reside in server format in a file named "Directory" inside of the 
directory whose contents they represent (with the addition of the header and status bytes 
as described above for ordinary files). For example, z:\Program Files\Microsoft Office 
would reside in c:\Cache\Program Files\Microsoft Office\Directory. This has the 
drawback of creating special file names that can't be used by files in the eStream volume, 
but again, for an early engineering implementation, this is an acceptable limitation. 

Another issue with deploying this implementation is that it is trivial to reverse-engineer 
this file format and copy files directly from the cache. 

Robust Implementation 

The cache will be organized into an index file and one or more cache data files. Multiple 
data files may be necessary as we may wish to allow the cache to grow larger than the 2 
GB file size limit (for some native file systems) or to span multiple drive letters on the 
client. The data files will only contain pages of file content. These pages will be aligned 
on page boundaries. The index file contains all the information needed to locate file 
pages, and is contained in a separate file for simplicity. 

Page and index files must reside on a local disk (rather than a network disk) and cannot 
be shared by multiple clients. 
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Each file with any pages currently resident in cache will have a data structure containing 
information about that file, including its file id, the file id of the directory containing it, 
the file's metadata, and the map for finding the file's data blocks. This data structure is 
very similar to the inode of a traditional file system, and will be referred to as the eStream 
inode. A naive implementation of the inode is described above; no doubt, we will want 
to reorganize this data structure for more compact representation and better performance. 
Note that one requirement of the inode is that it contain a status field for each page in the 
file. One character is sufficient for this status; whether or not we can make do with 
fewer than 8 bits is an open question. 



A hash table will be used to map file IDs to file inodes. 



Index hash table 



Collision chains of file inodes 



"too" inode 



"bar - inode 




The inode contains pointers to each block's location in one or more cache page files: 
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Foo's I node 



Page 1 



Page 2 



Page 3 



Page 4 
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Cache page file 1 Cache page file 2 
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To prevent race conditions, a single lock controls access to both the hash index and the 
linked list of requests that are pending network access. Individual pages in the cache may 
be locked for read or write access. Since each page's status is in the index, the index 
must be locked order to lock a page for reading or writing. The page states are controlled 
by the following state machine: 
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The dirty/clean distinction is between those pages that we have written locally (and thus 
cannot evict from the cache) and those pages that we haven't written (and thus can be 
refetched from the server). 

A page would be locked while it was being read or written for copying to the file system 
driver. The operation may thus proceed with the index unlocked, without the possibility 
of page eviction while a copy is still in progress. The FSD worker thread is the only 
thread that reads or writes pages from the cache, so it's the only thread that can lock or 
unlock these pages. The in flight state is only for pages that are currently being fetched, 
either as a demand fetch or as a prefetch. The prefetching thread is the only thread that 
will put pages into this state, and only the networking thread will move pages from in 
flight to unlocked. 

A list will be maintained of all "in-flight" requests. A single lock will control access to 
both this list and the cache index, so there are no race conditions between items being put 
on this list and data coming off the network. When the FSD worker thread gets a request, 
it acquires the index lock and looks at the status of the page. If the page is clean or dirty 
but unlocked, it will lock the page and copy it to the FSD. If the page is invalid, then this 
is a demand fetch, and the request is forwarded to the prefetcher. If the page is marked in 
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flight, then this is either a second request for an outstanding demand fetch, or it is a 
request for an in flight page. Either way, while this thread still holds the index lock, this 
request will be inserted into the list of in-flight requests. Race conditions might occur 
because the FSD might make multiple demand reads of the same page, or it may make a 
demand read to a page that is already in flight due to a prefetch. 

Reading requested pages off the network and writing them to the cache (and to the file 
system driver, if necessary), are where this race condition comes up. We need to ensure 
that a request for a page that has arrived does not end up in the list of "in flight" requests. 
The solution is the following: When a data page comes back from the server, the 
networking component acquires the index lock to find the cache location of this incoming 
page. If the page is not marked in flight in the cache, this is a bug. (Of course, this is a 
relatively benign bug, and the NW component could just ignore the page.) The 
networking thread leaves the page as marked in flight, however, and unlocks the index. It 
writes the incoming page into the proper location, but it saves the in-memory copy of the 
page. It then reacquires the index lock, marks the page as clean/unlocked (since it's now 
in its final location in the cache), removes each request in the in-flight list for this page, 
then releases the lock. (Any further requests for the same page will find the page 
clean/unlocked, so the FSD worker thread will be able to satisfy these requests directly.) 
The networking component then proceeds to satisfy all of the requests it pulled off the in- 
flight list by using the copy of the page that it saved in memory. This way, it doesn't 
have to lock the index the entire time it is sending completed requests to the FSD. 

Each of these complex scenarios is captured in the cache file's API's. As long as these 
are implemented correctly, other components don't need to worry about the exact 
sequence of operations that needs to occur. 

Free Space Management 

Free pages will be maintained as a free list in memory and as a bitmap on disk. The free 
list will be built from the bitmap on eStream client software startup. Access to the free 
list will be controlled by the same lock controlling access to the index. 

Evicting Cache Pages 

Individual cache pages may be evicted. There is an 8-bit field in the index for each 
page's importance. Initially, we will implement a random page replacement policy. 
Later, we will use this page importance field in an unspecified way to replace pages in 
such a way as to maximize interactive user performance and minimize application server 
load. Only clean/unlocked pages may be evicted. Pages that are evicted will eventually 
be put on the free list. Page eviction will only happen at "garbage collection" time. See 
"crash resilience and garbage collection," below. 

Handling Cache Size 

Growing the cache should not be an issue. The cache manipulation routines must know 
the overall size of the cache, in pages. Increasing the size of the cache on the fly should 
be a relatively straightforward process, as we merely need to lengthen the cache file(s) 
and add the new pages to the free page list. 
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Unfortunately, shrinking the cache is a much more difficult operation, since it potentially 
involves moving around pages that might currently be in use for paging operations or be 
in flight from the network. Changing the cache around at runtime is both difficult to 
implement correctly and a performance problem. The current plan is to support shrinking 
the cache only at eStream client software startup. The maximum allowed size of the 
cache will be stored in the Registry. On eStream client software startup, the current size 
of the cache will be compared against the allowed size specified in the registry; if it is 
larger than the maximum size specified in the registry, then the size of the cache will be 
reduced by evicting files and compacting the freed space. A request by the user to reduce 
the size of the cache will take effect the next time the client software starts. 

Note that files that the user writes to the z: drive are not considered candidates for 
eviction (unless the file is explicitly deleted.) This means that the user's on-disk cache 
may in fact grow to be larger than the limit they specify. 

Also note that at least one free page (not used by user- written files) is required for the file 
system to make forward progress. We also may want to require some minimal amount of 
cache before eStream will even run. Thus the maximum cache size specified by the user 
should be considered a "soft limit." There would be a "hard" minimum amount of space 
equal to the number of pages required to store the files written by the user on the z: drive 
plus a small amount of cache we designate just for running eStream. If this hard 
minimum is greater than the soft maximum specified by the user, the hard minimum 
would win. I would recommend preallocating and non-zero filling the file on disk so that 
we know that the space is available. 

Crash Resilience and Garbage Collection 

In order to provide crash resilience, the index will be periodically checkpointed to disk. 
Note that allocating blocks does not cause problems if the index is not updated. 
However, we cannot reuse a page's storage until that page has been marked free on disk. 

The solution to this problem is to periodically garbage-collect the cache (if it is nearly 
full), and writing the index to disk. The cache manager will alternate between writing 
two cache index files. The index file will have a marker at the end that indicates that it 
has been successfully written and a time stamp, and on startup the ECM will use the 
latest, fully written index. 

Data blocks will always be written directly to the cache page files. These files must be 
flushed before writing the index. 

Garbage collection involves the following steps: 

- lock the index 

- copy the free list 

- choose blocks in the cache to free, and make a list containing just the newly freed 
blocks. Mark these blocks as invalid in the file's inodes, but don't put them on the free 
list (yet) 
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- make a copy of the index 

- unlock the index 

- merge the list of newly freed blocks with the copy of the free list 

- flush all cache page files 

- write the new, merged free list (as a bitmap) and index to disk 

- lock the index 

- add the newly freed blocks to the free list 

- unlock the index 

- free any allocated data structures 

Index File Contents 

The index file contains the following items: 

- List of cache block files, with their sizes 

- Free block bitmap, per cache block file 

- Inodes for all files; may be stored hashed or may be rehashed on startup. 

Testing design 
Unit testing plans 

Cache file manipulation routines can be tested in isolation. We will write a standalone 
harness that exercises the functionality of the cache file manipulation routines by 
performing cache level operations directly. A multithreaded unit test for the cache 
manipulation routines would be ideal, so we can test the correctness and performance of 
our locking strategy without the need to build the entire cache manager. 

Each "thread" of execution described by this document can be separately tested by 
creating a testing harness providing that thread inputs and monitoring its outputs. 
Replacements for the EFSD interfaces can be very effective here. 

Stress testing plans 

An interesting stress test for the cache manager is if it can work correctly with very small 
caches, even all the way down to 1 page. (Or at least, a cache with all pages but one 
marked as dirty.) 

The cache manager will be able to operate in "verify mode," where requests that hit in the 
cache will still be sent to the server, and the pages returned by the server will be 
compared with the cached page's contents. 

The cache manager will support multiple different page checksum algorithms. We can 
use a fast algorithm for deployment while using a more rigorous one in development. 
This also has the benefit of allowing us to test the performance impact of various 
checksum algorithms. 
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The cache manager will have the ability to verify the integrity of the cache index and free 
page bitmap. In particular, it will have the ability to determine taht no pages are allocated 
to more than one file in the file system, and that each page belongs to a file or is on the 
free list. 

Stress testing for the ECM will include crash testing. 
Cache manager testing will include resizing the cache. 

Coverage testing plans 
Cross-component testing plans 

We can build a "cache only" file system by not using the prefetching and network 
components. This allows us to test the EFSD in conjunction with the cache manager 
without involving the prefetcher or the network component. 

Early implementation of the client will likely involve a null prefetcher that does no 
prefetching. 

We can use the testing harness for the cache manager that doesn't use the EFSD to drive 
the cache manager in conjunction with the prefetcher and network component. This 
allows us to test the combination of these components without driving it with the live file 
system driver. 

Upgrading/Supportability/Deployment design 

The client user-mode software and device drivers are packaged separately. (I.e. the client 
executable and the drivers are separate files on the disk.) This leads to the possibility of a 
"partial" upgrade that results in inconsistent versions of the drivers and client user-mode 
software. The drivers should support an interface that returns the version number of the 
driver, or of the interfaces provided by the driver. This will help the client software to 
recognize situations where it should tell the user to reinstall the client software and not 
result in bad system behavior. 

Most (all?) on-disk data files should have file headers containing at a minimum a magic 
cookie and the file format version number. This will help us with upgrades in the future. 



Open Issues 

We need to address what happens when a fetch is requested and no empty space can be 
found in the cache. The prefetcher should probably block until such time as space is 
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made available for this request. While operating with very small amounts of cache will 
obviously cause bad performance, it should not result in a deadlock. 



Omnishift Confidential 



Page 19 



^8 



eStream Cache Manager Straw Man Proposal 

Version 0.2 




Purpose 

The purpose of this document is to serve as the basis for the design of the eStream Cache 
Manager. As a straw man, this document is meant to serve as the basis for discussion, 
and anything here is subject to change. Assuming there are no major concerns with this 
document, I will proceed with producing a low level design for the cache manager. 

Requirements in Brief 

Support > 2GB client cache, possibly across multiple drives 

Provide some level of protection against piracy, via both the file system and the cache 
Fast lookup for what is in the cache and where to find it 
Support automatic and user-specified cache size policies 

As far as cache size goes, I think that it is reasonable for eStream 1 .0 for the cache to be 
limited to one disk partition and 2GB of space, but the design should allow for very large 
caches (spanning more than one file and possibly more than one drive letter.) Note that if 
the cache is greater than 2GB in size, it cannot be mapped into the address space of a 
single process under NT/2000 on x86. 

Cache Organization 

The cache will be contained in 2 or more files. One file will contain the cache indices, 
and one or more files will contain the data blocks for cached files. (More than one cache 
data file may be required if the cache is larger than the largest file allowed on the native 
file system.) This allows us to keep the cache index file memory mapped and only map 
the data file(s) if there is enough memory space to do so. 

Data Blocks 

The cache data file will contain data pages frome the file system 4k in size. 
Data will be stored in the cache uncompressed to allow easy page retrieval. 
Cache Index 

The cache index will be a b-tree. The key for the lookup will be the file id and page 
number requested. Keys in the b-tree are the set { volume #, file #, starting page, # of 
pages }. A lookup will succeed when the volume number and file number match, and the 
requested page is in the range from starting page to starting page + # of pages. The data 
stored for that key will be the offset into the cache for the beginning of the run. As is 
described in the file system proposal, the file number and starting pages are each 32 bits 
long. I propose making the starting page a 48 bit number and the number of pages a 16 
bit number. This allows us to have a very large total cache and reasonable sized runs of 
contiguous pages in the cache. 



Free space in the cache will have to be managed. Free blocks can be placed into a 
specially identified "free space file" in the index. Some auxiliary data structures may be 
convenient to make searching for a region of free space of a particular size. 

Metadata for a file would be stored in the cache. It would be indexed by page number -1 
in the index. 

Cache Replacement Policy 

For simplicity, I propose that the cache manager evict entire files from the cache when it 
decides that it needs to clear room in the cache. (Of course, any fragmentary file that is 
in the cache can be evicted.) We should implement LRU for cache replacement, so we 
will evict files for apps that have not been run recently. 

One Cache Per System 

Administrator priviledges are required to install eStream. While various users on a 
system might have conflicting desires about eStream configuration, such as the size of the 
cache, I think that it is reasonable to have a policy where the adminstrator controls the 
setup of the eStream client. By limiting the cache to one per system, we eliminate any 
ambiguity about cache use in a multiuser environment. 

Profiling and Prefetching 

Profiling and prefetching have been broken out as a separate component in the client. It 
will be described elsewhere. It is expected that while the profiler/prefetcher will want 
access to the cache data structures (i.e. it wants to know what's already in the cache), the 
logic associated with prefetching is not logically tied to the cache manager, and should 
thus be separated. 

Future Directions 

Compression of the cache could potentially be a big win. We could provide cache 
compression similar to the way that NTFS provides file compression - we compress some 
number of blocks at a time (e.g. 16) and only store the compressed data when it saves at 
least one block of storage. Caching of data on disk can sometimes be a performance win, 
since decompressing the data can be faster than trasferring it on disk if the disk is slow 
enough. 
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Functionality 

The eStream Windows NT/2000 File System Driver (EFSD) is a kernel-mode file system 
driver to which file requests will be forwarded by the NT I/O Manager. It is the point of 
contact for users to access files on an eStream server. It works with the NT File Cache 
Manager to insure that kernel file caching is available for eStreamed files. 

The Windows 98 EFSD is almost certainly to be very different from the driver for WNT 
and Win2K, and will not be described here. 

In this document, I'm assuming that the EFSD communicates closely with the eStream 
cache manager (ECM) to perform the various file system requests. There may in fact be 
several components— if for example the ECM is broken into sub-components. Also, this 
document assumes that the ECM is in user mode; if this ends up in kernel mode, we will 
need significant changes to the interfaces to it. 

Data type definitions 

File handle 

A file handle passed between the EFSD and the ECM is defined by the ECM: 
typedef uint32 EFSDFileHandle ; 

Names 

All file and directory names will be passed as counted Unicode strings, basically as de- 
fined by the NT header files. Note, however, that in NT the Buffer field is a pointer; for 
our purposes in communicating with the ECM, it's a NULL-terminated variable length 
array: 

typedef struct _UNICODE_STRING { 
USHORT Length; 
USHORT MaximumLength; 

USHORT Buffer [1] ; // NULL-terminated, 2 -byte 

// characters 

} UNICODE_STRING; 
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Time stamps 

The NT standard time format is a signed 8-byte integer representing the number of 100- 
nanosecond intervals since ^^£0Pfc These time stamps will be tracked for files 
and directories: 

□ Creation time 

□ Modification time 

File attributes 

File attributes are contained in an unsigned 4-byte integer. This subset of attributes from 
Windows NT will be supported: 

F I LE_ATTR I BUTE_READONLY 
F I LE_ATTR I BUTE_D I RECTORY 
F I LE_ATTRI BUTE_ARCHI VE 
F I LE_ATTR I BUTEJTORMAL 
F I LE__ATTR I BUTE ^TEMPORARY 

These attributes are not supported: 

F I LE_ATTR I BUTE_H I DDEN 

FILE_ATTRIBUTE_SYSTEM 

F I LE_ATTR I BUTE_DEVI CE 

F I LE_ATTR I BUTE__S PARSE_F I LE 

F I LE__ATTR I BUTE__RE PARS E__PO I NT 

F I LE_ATTR I BUTE_COMPRE S SED 

F I LE_ATTRIBUTE_OFFLINE 

F I LE_ATTR I BUTE_NOT_CONTENT_ I NDEXED 

FILE_ATTRIBUTE_ENCRYPTED 

File size 

File size will be represented as a 4-byte unsigned integer. Since sparse files are not sup- 
ported, there will only be one file size passed between the ECM and the EFSD. 

Metadata 

This structure is defined to pass file and directory metadata between the EFSD and the 
ECM: 

typedef struct { // 24 bytes, 4-byte aligned 
int64 CreateTime; 
int64 ModifyTime; 
uint32 FileSize; 
uint32 Attributes; 
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} Metadata; 

Interface definitions 

The EFSD is called by several different components, including 

□ the NT Executive (I/O Manager, Virtual Memory Manager), for standard file sys- 
tem requests 

□ the ECM, for these same file system requests, and to invalidate cached pages for 
coherency 

□ the client start software, to start and stop the EFSD 

The EFSD supports standard FSD interfaces to the NT Executive modules; not all possi- 
ble interfaces are supported, because the eStream file system is relatively low- 
fimctionality (compared to NTFS, for example). 

The following file system requests will be supported; the interfaces for them will not be 
shown here, as they can be found in the DDK documentation. 

□ Create IRPs, for both new and existing files 

□ Cleanup and Close IRPs 

□ Read and Write IRPs: 

o synchronous and asynchronous 
o cached and non-cached 
o paging and non-paging 

□ Fast I/O reads and writes (with buffers or MDLs) 

□ File information (get and set) IRPs 

□ Directory query IRPs 

□ Volume information (get and set) IRPs 

□ File system information (get and set) IRPs 

□ Flush buffer IRPs 

□ System shutdown IRPs 

□ Various Fast I/O queries 

The EFSD will not handle Directory Notification IRPs, nor will it support hard links 
(which are supported natively on NTFS on W2000 only): neither of these requests are 
required, and no expected user functionality will be lost without them. We are presently 
not supporting byte-locks; this may need to be revisited if the need arises. 

In addition to the interfaces to the NT Executive, the EFSD will support various inter- 
faces from other client components; all these will be sent via IOCTL calls. The first ones 
listed are simple support interfaces; the interfaces between the ECM and the EFSD fol- 
low these. 

An IOCTL coming in to the kernel— via a DeviceloControlQ call— has the following pa- 
rameters: 
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□ IOCTL control code 

□ input buffer pointer 

□ input buffer size 

□ output buffer pointer 

□ output buffer size 

□ pointer to a 4-byte variable to receive the number of bytes written to the output 
buffer 

□ pointer to an OVERLAPPED structure for asynchronous operation (always 
should be NULL for EFSD) 

All of the following interfaces are described in terms of the IOCTL buffers sent and re- 
ceived for each control code. 

The following interfaces are called from the controlling client component (StartClient). 

Starting and stopping the file system 

The eStream FSD will be loaded into the kernel when a system is rebooted; i.e., it is al- 
ways resident. If applications access files on this FSD via a drive letter, then the file sys- 
tem is implicitly turned off while a symbolic link for that drive letter is not present. Even 
when a drive letter symlink exists, the EFSD will not accept requests until the START 
IOCTL is sent. 

These IOCTL control codes will be defined for starting and stopping the eStream FSD: 



IOCTL_EFS_START_FS 
IOCTL_EFS_STOP_FS 

Starting the FSD 

The input buffer for the START IOCTL should have the following: 

□ version id: 4-byte identifier for the client component 

□ debug flags: 4-byte value indicating the debug level to use 

The output buffer for this IOCTL will be filled with the following: 

□ version id: 4-byte identifier for the EFSD version present 

□ status: 4-byte value, with one of the following: 

E F S_S TATUS_SUCCE S S 
EFS_STATUS_BAD_VERSION 
EFS_STATUS_BUFFER_TOO_SMALL 
EFS_STATUS_DUPLICATE_REQUEST 
EFS STATUS ABNORMAL__TERM I NAT I ON 
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The status return value from this IOCTL will be one of the following: 

□ STATUS_SUCCESS 

□ STATUS_INVALID_DEVICEJREQUEST 

A DUPLICATE REQUEST error is returned if the FSD is already started. 
Stopping the FSD 

The input buffer for the STOP IOCTL should have the following: 

□ force: 4-byte value 

o 0: shutdown only if no outstanding files are open 
o 1 : shutdown regardless of state of open files 

The output buffer for this IOCTL will be filled with the following: 

□ status: 4-byte value, with one of the following: 

EFS_STATUS_SUCCESS 
EFS_STATUS_BUFFER_TOO_SMALL 
E F S _J3 T ATU S_DU P L I CAT E_RE QUE S T 
EF S_STATUS_ABNORMAL_TERMI NAT I ON 

The status return value from this IOCTL will be one of the following: 

□ STATUS_SUCCESS 

□ STATUS JNVALrojDEVICE_REQUEST 

A DUPLICATE REQUEST error is returned if the FSD is already stopped. 

Cache management interfaces 

The following two interfaces are defined for use by the ECM to potentially invalidate 
data in the NT File Cache. 

These IOCTL control codes will be defined for cache management for the eStream FSD: 

I OCTL_EFS_INVALIDATE_FI LE 

I OCTL_EFS_INVAL IDATE JD I R_CONTENTS 

Invalidating a f ile 

The input buffer for the INVALID ATEJFILE IOCTL should have the following: 

□ handle: 4-byte EFSDFileHandle for the open file that must be invalidated 
Omnishift Technologies, Inc. 5 Company Confidential 
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The output buffer for this IOCTL will be filled with the following: 
□ status: 4-byte value, with one of the following: 



EFS 
EFS 
EFS 
EFS 



STATUS 
STATUS 
STATUS 
STATUS 



SUCCESS 

BUFFER_TOO_SMALL 
FI LE_NOT_OPEN 
ABNORMAL TERMINATION 



The status return value from this IOCTL will be one of the following: 

□ STATUS_SUCCESS 

□ STATUSMVALEDDEVICEREQUEST 

If in fact the file is open, but not present in the NT File Cache, this IOCTL will simply 
succeed; no error is returned. 

Invalidating directory contents 

The input buffer for the INVALID ATE_DIR_CONTENTS IOCTL should have the fol- 



□ handle: 4-byte EFSDFileHandle for the open directory whose contents must be 
invalidated 

The output buffer for this IOCTL will be filled with the following: 

□ status: 4-byte value, with one of the following: 



The status return value from this IOCTL will be one of the following: 

□ STATUS_SUCCESS 

□ STATUS JOWALID_DEVICE_REQUEST 

General file system requests 

All file system requests that cannot be completely handled by the EFSD will be passed on 
to the ECM. Since the ECM is likely to be a user-mode service, the EFSD cannot call it 
directly; thus these "calls" are made by having the ECM send lOCTLs to the EFSD to get 
and fulfill requests. Each file system request requires multiple IOCTLs sent from the 
ECM to the EFSD: 

Omnishift Technologies, Inc. 6 Company Confidential 
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EFS 
EFS 
EFS 
EFS 



STATUS 
STATUS 
STATUS 
STATUS 



SUCCESS 

BUFFER_TOO_SMALL 
F I LE_NOT_OPEN 
ABNORMAL TERMINATION 



eStream File System Driver Low Level Design 

1 . The ECM sends an IOCTL to the EFSD to get the next request 

2. The ECM sends a second and/or third IOCTL to finish the request 

The following IOCTL control codes will be defined by the EFSD for use by the ECM: 

IOCTL_EFS_GET_REQUEST 

IOCTL_EFS_RETRY_REQUEST 

I OCTL_EFS_GET_CREATE_NAME 

IOCTL_EFS_FINI SH_CREATE 

IOCTL_EFS_FINISH_CLOSE 

I OCTL_EFS_F INI SH_READ 

I OCTL_EFS_GET_WR I TE_DATA 

IOCTL_EFS_FINISH_WRITE 

IOCTL_EFS_GET_RENAME_TARGET 

I OCTL_EFS_F INI SH_RENAME 

IOCTL_EFS_FINISH_DELETE 

I OCTL_EFS_FI NI SH_METADATA_READ 

IOCTL_EFS_FINISH_METADATA_WRITE 

For the DeviceloControlO call sending IOCTL_EFS_GET_REQUEST, these parameters 
are invariant: 

□ the 10 control code will be IOCTL_EFS_GET_REQUEST 

□ input buffer pointer will be NULL 

□ input buffer size will be 0 

□ output buffer must be non-NULL 

□ output buffer size must be at least 40 bytes— this is the largest buffer needed for 
any request (subject, of course, to slight modifications) 

□ pointer to bytes returned will be non-NULL 

□ overlapped pointer will be NULL 

The IOCTL_EFS_RETRY_REQUEST is sent by the ECM (or some other user-space cli- 
ent component) to tell the EFSD that, yes, it needs to delete all intermediate information 
about a request already sent back with a GET_REQUEST call, and put the request back 
on the list for the ECM to retrieve. This eases implementation issues for the ECM. The 
input buffer for a RETRY_REQUEST is: 

□ request id of the previously retrieved request 
There is no output buffer for a retry request IOCTL. 

What follows is a list of file system requests from the NT I/O Manager, and the IOCTL 
calls needed from the ECM to service those requests. For all cases, if the EFSD writes to 
the output buffer for an IOCTL, the "bytes returned" field is written with the number of 
bytes written. 
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Create 



This is used for both create and open, for files and directories. 



GET REQUEST 



The output buffer for the IOCTL_EFS_GET_REQUEST will be filled with the follow- 



□ retry count: 4 bytes— how many retries this GET_REQUEST corresponds to. 
First time, this is 0. 

□ flags: 4 bytes, one or more of the following ORed together 

o CRE ATE_ONLY : fail if file exists already 

o OPEN_ONLY: fail if file does not exist already 

o TRUNCATE: overwrite existing file 

o DIRECTORY: create a directory 

o FILE: create a plain file 

o DELETE_ON_CLOSE: delete file on last close 

o IGNORE_CASE: obvious 

□ permissions: 4 bytes, one or more of the following ORed together 

o READ 
o WRITE 
o EXECUTE 

□ length of filename: 4 bytes, specifying the byte size needed for the Unicode string 
sent in the next call 

Total size of output buffer: 24 bytes 

GET_CREATE_NAME 

The input buffer for this IOCTL should have the following data: 

a request id: the id sent in the previous call 
The output buffer for this IOCTL will be filled with the following information: 

□ request id for this transaction 

□ fully qualified name as a counted Unicode string (including drive letter, if any): 
the length needed was sent back in the GETJREQUEST call 

FINISH_CREATE 

The input buffer for this call should have the input buffer filled as follows: 



ing: 



□ 



□ 



type: a 4 byte field that indicates a Create request 

request id: a 4 byte field that will be subsequently sent in the calls to match this 
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□ request id: the matching id from the GETJREQUEST call 

□ status: the NTSTATUS result from this request 

□ handle: the 4-byte handle for this opened file, that can be used for subsequent file 
system requests. A unique value will indicate a bad handle, and a failed Create 

□ a Metadata buffer: the metadata for the created/opened file/directory. 

The output buffer for this IOCTL should be NULL. 

Note that a TRUNCATE Create request should cause the metadata sent back to reflect the 
possibly new (zero) length. 

Close 

This closes a handle of a previously opened file or directory. The EFSD will optionally 
send the updated metadata for this file in the GETJREQUEST output buffer. If the file 
has been modified in any way, the metadata fields will be non-zero; else they will all be 
zero. 

GETJREQUEST 

The output buffer for this call will be filled with the following: 

□ type: a 4 byte field that indicates a Close request 

□ request id: 4 bytes, for use in subsequent calls for this request 

□ retry count: 4 bytes — how many retries this GETREQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes, the handle for the previously opened file 

□ metadata for this file/directory: 24 bytes 

o creation time stamp 

o modification time stamp 

o file/directory size in bytes 

o attributes (as described above) 

Total size of output buffer: 40 bytes 
FINISH^CLOSE 

The input buffer for this call should contain the following: 

□ request id for this transaction 

□ status: the NTSTATUS for this request 

Read 

This is used for reading file data. 



GETJREQUEST 
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The output buffer for the IOCTL_EFS_GET_REQUEST will be filled with the follow- 



□ type: a 4 byte field that indicates a Read request 

a request id: a 4 byte field that will be subsequently sent in the IOCTL to match this 
request 

□ retry count: 4 bytes— how many retries this GET_REQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes, the handle for this previously opened file 

□ offset: 4 bytes, the file offset, in bytes, to read from 

□ length: 4 bytes, the length of the read, in bytes 

NOTE: The buffer requested in the (offset, length) pair will not cross a 4K page bound- 



Total size of output buffer: 24 bytes. 
FINISH^READ 

The input buffer for this call should have the input buffer filled as follows: 

□ request id: the matching id from the GET REQUEST call 

□ status: the NTSTATUS result from this request 
o the number of bytes successfully read; 0 on error 

□ the data from the read; not present on error 

The output buffer for this IOCTL should be NULL. 
Write 

This is used for writing file data. 
GET_REQUEST 

The output buffer for this will be filled with the following: 

□ type: a 4 byte field that indicates a Write request 

□ request id: a 4 byte field that will be subsequently sent in matching calls for this 
request 

□ retry count: 4 bytes— how many retries this GET_REQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes, the handle for this previously opened file 

□ offset: 4 bytes, the file offset, in bytes, to write to 

□ length: 4 bytes, the length of the write, in bytes 

□ file length: 4 bytes, the length the file will be if this write succeeds 
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Total size of output buffer: 28 bytes. 

NOTE: The buffer requested in the (offset, length) pair will not cross a 4K page bound- 



GET__WRITE_DATA 

This IOCTL will have an input buffer with: 

□ request id for this transaction 

□ status: if not STATUS_SUCCESS, this ends the request; the output buffer is un- 
touched, and no FINISH_WR1TE call is expected. 

And the output buffer will be filled with: 

□ request id 

□ data buffer for the write— the byte length sent in the previous GET_REQUEST 
FINISHJAfRITE 

For this finishing request IOCTL, the input buffer has these contents: 

□ request id: the matching id from the GETREQUEST call 

□ status: the NTSTATUS result from this request 

□ bytes actually written; should be equal to requested bytes unless failure occurs. 
Rename 

This is used for renaming a file or directory. 
GET_REQUEST 

The output buffer for this IOCTL will be filled with the following: 

□ type: a 4 byte field thai indicates a Rename request 

□ request id: a 4 byte field that will be subsequently sent in the FINISH REQUEST 
call to match this request 

□ retry count: 4 bytes — how many retries this GET REQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes; the handle for this previously opened file or directory 

□ length of target name: 4 bytes; the byte length needed for a counted Unicode 
string for the target name 

Total size of output buffer: 20 bytes. 

GET^RENAMEJTARGET 
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The input buffer for this call will have the following: 

□ request id for this transaction 

□ status: if not STATUS_SUCCESS, then this terminates the request: the output 
buffer is not touched, and no FINISHRENAME call should be sent 

The output buffer will be filled with the following: 

□ request id 

□ target name: a counted Unicode string, using the same number of bytes as sent in 
the GET REQUEST output buffer 

FINISH_RENAME 

The input buffer for this call should have the following: 

□ request id 

□ status: NTSTATUS for the transaction 
Delete 

This is used for deleting a file or directory. 
GET_REQUEST 

The output buffer for this call will be filled with the following: 

□ type: a 4 byte field that indicates a Delete request 

□ request id: a 4 byte field that will be subsequently sent in the call to match this re- 
quest 

□ retry count: 4 bytes — how many retries this GETREQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes; the handle for this previously opened file or directory 

Total size of output buffer: 16 bytes. 
FINISH^DELETE 

The output buffer for this call should be NULL. 
The input buffer should have the following contents: 

□ request id: matching id from the GETREQUEST call 

□ status: NTSTATUS of this request 
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Metadata read 

This is used for requesting metadata about a file or directory. 
GET^REQUEST 

The output buffer for this call will be filled with the following: 
type: a 4 byte field that indicates a Metadata request 

request id: a 4 byte field that will be subsequently sent in the call to match this re- 
quest 

retry count: 4 bytes— how many retries this GET_REQUEST corresponds to. 
First time, this is 0. 

handle: 4 bytes; the handle for this previously opened file or directory 
Total size of output buffer: 16 bytes. 
FINISH JMETADA TAJREAD 
The output buffer for this IOCTL will be NULL. 
The input buffer should have the following contents: 

□ request id: id from the corresponding GET_REQUEST 

□ status: NTSTATUS for this operation 

□ the following data about the file or directory: 

o creation time stamp 

o modification time stamp 

o file/directory size in bytes 

o attributes (as described above) 

Metadata write 

This is used for setting metadata for a file or directory. 
GET^REQUEST 

The output buffer for this call will be filled with the following: 

□ type: a 4 byte field that indicates a Metadata Write request 

□ request id: a 4 byte field that will be subsequently used for all calls for this request 

□ retry count: 4 bytes — how many retries this GETREQUEST corresponds to. 
First time, this is 0. 

□ handle: 4 bytes; the handle for this previously opened file or directory 

□ metadata for this file/directory: 24 bytes 

o creation time stamp 

o modification time stamp 



□ 
□ 

□ 

□ 
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o file/directory size in bytes 
o attributes (as described above) 

Total size of output buffer: 40 bytes. 

FINISH_METADATA_WRITE 

The input buffer should have the following contents: 

□ request id: from the previous call 

□ status: NTSTATUS for this request 

Component design 

This section is organized in the following manner: 

1 . General layout of the eStream file system driver 

2. General observations about the low level design 

3. Organization of data structures 

4. Description of the algorithms for communication with the ECM 

5. Description of each dispatch routine 

Layout 

The EFSD will be generally organized in the following manner: 

□ All major IRPs will have their own dispatch routine. 

□ All actual I/O requests to the ECM will be generalized from the dispatch routines 
to a set of routines that handle the communication with the ECM, to isolate this 
aspect. 

□ All utility functions will be in their own file or files. 

General points 

The design of the EFSD will look a lot like the sample FSD from Rajeev Nagar's NT FS 
Internals book, which looks a whole lot like the Fastfat FSD source from the NT IFS kit. 

Here is a list of general points that can be made about the EFSD: 

□ Any IRP that can be handled asynchronously will be posted to a work queue; this 
means that the dispatch routine for such an IRP must be able to handle being 
called in a context other than the original requestor. 

□ There are no volumes, and no Volume Parameter Block or Volume Control 
Block. There isn't a VPB for a network redirector; I've verified this with the 
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□ 



LanManager redirector. Hence we don't have to support any operations on a 
volume in EFS. 

We will not allow the creation of paging files in EFS. There is a bit available for 
a Create IRP that specifies this, and we can complete the TJRP with an unimple- 
mented error return code. 

All file synchronization will be on a File Control Block (FCB) basis, using the 
standard Resource and PagingloResource ERESOURCE objects used by the rest 
of the Windows Executive. 

o User requests will be synchronized by acquiring the main Resource- 
shared for reads, shared for most writes, and exclusive for file size 
changes, deletion, etc. 
o Paging I/O requests will be synchronized by acquiring the PagingloRe- 
source— again, shared for reads, shared for most writes. Exclusive access 
will be needed to set file sizes. 
Most disk file systems have a resource associated with a VCB, which is acquired 
exclusively for creation/deletion etc. We will have a global EFS resource for this, 
since there are no VCBs. 

Asynchronous requests will be handled by posting the IRP to the Cntical- 

WorkQueue, and marking the IRP as pending. 

o A common worker routine will be used for all async posts, which will dis- 
patch the IRP to the appropriate real IRP routine when it's invoked, 
o An async request will be defined as one that IoIsOperationSynchronousQ 
returns false, and the EFSD is the top-level component (see below) 
The EFSD will track the top-level IRP for the thread whose context it is running 
in. In particular, 

o No async processing request will be honored unless the EFSD is the top- 
level component 

o No cache manager requests will be made unless the EFSD is the top-level 
component 

o EndOfFile size— that is, the true size of the file— will not be extended or 
changed by paging I/O 
EFS will not support holes in files, and hence the ValidDataLength FCB field will 
be set to disable this. This means the AllocationSize for an eStream file/directory 
will always be equal to the EOF size. 

Most fast I/O routines will be supported in EFS. We will use the FSRTL supplied 
routines for fast reads and writes. 

All. cache manager resource acquire/release callbacks will be supported. All will 
point to common routines that simply acquire or release the main or paging I/O 
resource for the FCB. The Context pointer passed into all of them will be the 
FCB for the stream. 

i Synchronous read/write requests will update the CurrentByteOffset in the File ob- 
ject. 

i Each Create will result in a unique Context Control Block (CCB) data structure; 
this will be small, and only hold those few fields needed: 

o For the Directory Control IRP, a CCB needs to hold the current entry in- 
dex and the pattern originally used— for subsequent queries 
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o A field for various flags 
a A single FCB will represent all current open instances of a file. When a Create 
request comes in, the EFSD will search the current open FCBs to try to find one 
matching this file/directory name. 

o For now, this will be a hash table on the file name. We can improve this 
as needed. 

o The EFS global resource must be acquired exclusively: 

■ before the global FCB data structure is searched. Why? If it's 
just a read, can't we acquire it non-exclusively? 

■ before a new FCB is added to the list 

■ before an FCB is deleted from the list 

□ EFS will not support open by file ID; hence the Filelnternallnformation class for a 
File Information IRP will not be supported. 

□ Actual I/O will be directed to standard routines in a separate file, so they can be 
isolated and updated easily as our method of transferring data changes. 

□ Here's how to do file/directory renames: 

o The I/O manager will send to the EFSD this sequence: 

■ Create for source 

- Create for target, with the SL_OPEN_TARGETJDIRECTORY 
flag set 

■ Set Information with a Rename request for the source, sending the 
target directory FileObject handle and the target name in the 
FilelnformationClass record. 

o EFSD needs to do this: 

■ When it receives the Create for the target and the target directory 
exists, return STATUS_SUCCESS 3 and change the name in the 
FileObject to the basename of the target (the full pathname of the 
target is sent in), and set the Status.Information to FILE EXISTS 
or FILE DOES NOT EXIST, as appropriate. If the target directory 
doesn't even exist, return PATH NOT FOUND. 

■ When it receives the Set Info request, if all the flags check out 
(e.g., if the file exists, ReplaceExisting must be TRUE), send a Re- 
name request to the ECM. 

□ Reads and writes to only regular files will be supported, not to directories. 

□ Any code that touches user buffers or can call routines that may throw exceptions 
must be guarded by a try/except block. 

□ Some tips on memory allocation (from /perforce-doc-dir/osrdocs/defensive- 
driv.html) 

o We will use our own memory allocation/deallocation routines, instead of 

ExAllocatePool() et al. directly 
o These routines can do various checks for trashing memory: 

■ fill allocated memory with a pre-defined bit pattern, instead of ze- 
roes; fill deallocated memory with a different pattern. 

■ allocate a header/trailer with standard information, like where allo- 
cated, from what pool, etc. 
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■ change the bit pattern in the header/trailer on deallocation, and 
look for freeing memory twice 
o We probably want to allocate using lookaside lists, since we'll be allocat- 
ing and deallocating smallish chunks of memory for our data structures. 

Data structures 

The following are major data structures used internally by the EFSD. Data structures 
used to communicate with the ECM are described in the following section. 

Nodeldentifier 

A Nodeldentifier is a simple structure that starts all other structures used in the EFSD. 
This makes a good debugging check to insure that we receive and are operating on the 
right type of data. It consists of two fields: an identifier field, and the total structure size. 

FCB 

The FCB is a critical data structure for the file system driver. There is one FCB structure 
allocated for each unique file or directory that is currently open-regardless of how many 
open handles there are for this entry. Multiple CCB structures can point to a single FCB. 

Logically at least, part of an FCB is exposed to the Cache Manager and VMM to support 
caching and paging I/O. We will follow the example of Rajeev Nagar's book, and embed 
the FSRTL_COMMON_FCB_HEADER and other required structures directly in an 
FCB. 

Here are the basic contents of an EFSD FCB: 

□ The required FCB contents above 

□ An open handle count: incremented on Create, decremented on Cleanup 

□ A reference count: incremented on Create, decremented on Close. The semantics 
of NT file requests require these two be used together to determine when an FCB 
can be deleted 

□ A pointer to the next FCB on its hash list 

□ A pointer to the first CCB opened for this FCB 

□ The fully qualified name of the file/directory 

□ The Metadata associated with this file/directory 

□ A SHARE ACCESS structure used to check sharing violations 



□ Various flag bits 
CCB 



A CCB represents a currently opened handle to a file or directory. If two processes have 
the same file opened, there will be a unique CCB allocated for each process. 
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Here are the basic contents of an EFSD CCB: 



□ A pointer to its corresponding FCB 

□ A pointer to the next CCB opened for its FCB 

□ A pointer to the FileObject opened for this file/directory 

□ The current file index for a directory query 

□ The directory search pattern used for directory queries for this opened handle 

□ Various flag bits needed 

IrpContext 

An IrpContext structure encapsulates the interesting data from an IRP, and the current 
stack location, for easy access. This structure is allocated on entry to dispatch routines, 
and used during processing, before being deallocated on exit. 



Communicating with the ECM 

I tend to divide a file system driver into two logical parts: 

1 . A front-end that understands the NT FSD interfaces and semantics 

2. A back-end that actually perform the requested actions 

Our back-end is the (admittedly ugly) interface for communicating with the ECM, which 
currently sits in user-space. It's very important to design the ECM such that its front-end 
and back-end are nicely separated: since the ECM may move to kernel space, or we 
might find better interfaces for them to communicate with each other, we need to design 
with this in mind. 

Given the current ECM interfaces defined above, here is a basic design to handle them: 



□ An EfsdRequest object will be created for each request that must be handled by 
the ECM. 

□ Each dispatch routine that results in a request to the ECM will allocate an 
EfsdRequest and send it to a common routine for further processing. 

□ New requests will be placed in a NewRequest queue. 

□ Requests that have been "sent" to the ECM, but not yet finished, will be removed 
from the NewRequest queue and placed in a PendingRequest list. 

□ Finishing a request entails removing it from the PendingRequest list, returning the 
contents to the dispatch routine, and destroying the request object. 



Data structures 



EfsdRequest 

This contains: 
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request id: a number that uniquely identifies this request 
type: the type of request (e.g., Create, Write) 
FCB pointer for the file/directory for this request 
IrpContext pointer for this request 

a kernel event object used for signaling that the request is satisfied 
NewRequestSemaphore 

The EFSD device object's device extension will contain a semaphore dispatcher object, 
initialized to non-signaled, and with an initial limit ofMAX LONG. When a request is 
added to the NewRequest queue, this semaphore is released (and the count is incremented 
by 1); the GET_REQUEST IOCTL will wait on this semaphore object (which decre- 
ments the count by 1 ). 

NewRequestQueue 

This is actually a kernel-managed interlocked list that is allocated in the device extension 
area, guarded by a spin lock that's also located in the device extension. Requests will be 
added to the tail, retrieved from the head. 

PendingRequestList 

This list must be searched when an ECM non-GET REQUEST IOCTL is received, so we 
can't use an interlocked list. We'll use a global single-linked list structure, with elements 
allocated from a dedicated lookaside list using non-paged memory. Before a thread can 
access the list elements, it must acquire a mutex. 

Algorithm 

□ All dispatch routines, and asynchronous read/write routines, will call LowLevel- 
PostRequest() to have their requests satisfied. LowLevelPostRequest() is itself 
synchronous; that is, the code calling it is something like this: 

status = LowLevelPostRequest (fcb, irp_context) ; 
/ / we ' re done ; 

// do all deallocation and cleanup needed 
IoCompleteRequest (irp, . . . ) ; 

□ LowLevelPostRequestQ will do the following: 

if this is a read or write IrpContext 

see how many requests are needed to satisfy the IRP: can't span page boundary 
allocate the N requests 

allocate an array of N event pointers to hold the request events 
assign the pointers to the array 
if > THREAD_WAIT_OBJECTS 

allocate an array of N PKWAIT_BLOCKS 



□ 
□ 
□ 
□ 
□ 
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else if this is a directory query IrpContext 
look at the FileSize of the directory FCB 

allocate enough read requests to read all directory data from the ECM 
as above, allocate an array of event pointers, wait objects (if necessary) 

else 

allocate a new EfsdRequest for the incoming FCB and IrpContext 
place all requests on the NewRequestQueue, using ExlnterlockedlnsertTailList() 
call KeReleaseSemaphore() on the NewRequestSemaphore 
if multiple requests generated 

call KeWaitForMultipleObjectsO on the request object's event 

else 

do a KeWaitForSingleObject() on the request object's event 
when the event(s) is/are signaled 
fill in the values into the IRP 
deallocate the EfsdRequest(s) 
return status 

□ When a GET REQUEST IOCTL comes in from the ECM, the EFSD will do the 
following: 

do a KeWaitForSingleObject() on the NewRequestSemaphore 
remove the first request from the NewRequestQueue, 

using ExlnterlockedRemoveHeadList() 
lock the PendingRequestList mutex 
place this request on this list 
release the mutex 

fill in the IOCTL output buffer identifying the request 
complete the IOCTL IRP 

□ When a RETRY REQUEST IOCTL is received, the following takes place: 

lock the PendingRequestList mutex 
search the list by request id; error if not found 
remove the request from the list 
release the mutex 

enqueue the request on the NewRequestQueue— on the list head 
release the NewRequestSemaphore 
complete the IOCTL IRP 

□ When any "finishing" IOCTL is received— i.e,, the second of two or the third of 
three — the following is done: 

acquire the PendingRequestList mutex 

search the list by request id; error if not found 

remove the request from the list 

release the mutex 

do all buffer copying, set all flags, 

and otherwise insure the input IRP has the correct state 
signal the EfsdRequest event 
complete the IOCTL IRP 

□ When any other request IOCTL is received— e.g., the second of three— this is 
done: 
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acquire the PendingRequestList mutex 
search the list by request id; error if not found 
release the mutex 

fill in the output buffer of the IOCTL as appropriate 
complete the IOCTL IRP 

Dispatch routines 
DriverEntry 

This does a whole slew of initialization, including the dispatch table, fast I/O table, the 
cache callbacks, the FCB hash table and its synchronization object, creates the FS device 
object, and sets up the interface with the ECM. 

Create 

There is one Create routine; there will be no async processing of Create requests. Ulti- 
mately, its job is to send a create request to the ECM, and return SUCCESS or not to its 
caller. Here is a general algorithm for this routine. 

create an IrpContext 
if a page file is requested 
return error 

generate the absolute pathname of the requested file: 
if OPEN_TARGET_DIRECTORY specified 
if OPEN ONLY not specified 
return error 

generate the pathname of the parent directory of the requested file 
if a related file object is specified 
if the related file is not a directory 

return error 
if the related filename doesn't start with T 

return error 
if the input filename starts with T 

return error 

concatenate the input filename with the related file directory 
else use the input filename 

if the input filename does not start with T } 
return error 
acquire the global EFS resource exclusively 
search the FCB hash table for this file by name 
if not found 

call LowLevelPostRequest() to send the request to the ECM 
if error is returned from ECM 

return the correct error: FILE NOT FOUND or PATH NOT FOUND 
create a new FCB and add to the hash table 
call loSetShareAccess() for this FCB 

else 

check input attributes/flags against those in FCB 
if opening for write or delete on close 
call MmFlushlmageSection() 
if this fails 
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return error 

if failing mismatch— e.g., if loCheckShareAccess() fails 
return error 
create a new CCB for this file 
set all appropriate flags on CCB and/or FCB 

COMMONFCBHEADER flag is set to FastlolsPossible 
set all fields on input FileObject: 

write through flag 

FsContext points to common FCB header 
FsContext2 points to CCB 
if OPEN_TARGET_DIRECTORY is specified 
search for the input target object: 

look for this in the FCB hash table 

if not found 

send a Create request for this file to the ECM 
if this target filename exists 

set Status.lnformation to FILE_EXISTS 

else 

set Status.lnformation to FILE_DOES_NOT_EXIST 
if a Create was sent to the ECM for the input FileObject 
send a Close request for the input target to the ECM 
change the name in the FileObject to the basename of this file 
the CCB and FCB remain opened for the target directory 
all necessary data structures should be deallocated, for success and error 
release the global EFS resource 

Cleanup 

No async posting of Cleanup requests will be done. Algorithm: 

acquire the global EFS resource, and the FCB Resource, for this Hie, exclusively 
if this file is marked for deletion 

if this is the last open handle for the file 

acquire the PagingloResource exclusively 

set the file size in the FCB to 0 

release the PagingloResource 

purge the cache, if necessary, with MmFlushlmageSection() 
call LowLevelPostRequest() to send a Delete request to the ECM 

decrement the count of open handles in the FCB 

if caching is on 

flush the cache by calling CcUnitializeCacheMaps() 

any time stamps must be updated if accesses were done using fast I/O 

set the FO_CLEANUP_COMPLETE flag in the FileObject 

call loRemoveShareAccess() 

release the global EFS and FCB Resources 

Close 

There will be no async posting of Close requests. Note that we only send a Close request 
to the ECM if this is the last close for an open file— i.e., we're matching Close requests 
with Create requests. 

Algorithm: 
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acquire the global EFS resource exclusively and the FCB Resource 
deallocate the CCB 

decrement the reference count for the FCB 
If the FCB ref count is now 0 

remove the FCB from the hash table 

deallocate the FCB 

call LowLevelPostRequest() to send a Close request to the ECM, 

updating metadata if necessary 
release the global EFS resource and the FCB Resource 

Read 

Reads will definitely be open to async posting. A general algorithm: 

if the read length is 0 

return success 
if the target file object is a directory 

return error 

if this IRP can be handled async 
post for async processing 

if this read is non-buffered, and it's not for paging I/O, and 

there is a mapped data section object for this file 
acquire the FCB main Resource exclusive, and the PagingloResource shared 
call CcCacheFlush() on the range of this read (current byte offset, length) 
release the FCB resources 
if this is for paging I/O 

. acquire the PagingloResource shared 
else 

acquire the main Resource shared 
if this read starts beyond EOF 

return EOF 
if the read length goes beyond EOF 

truncate the length 
if this is a buffered read 

if the PrivateCacheMap is NULL 
call CclnitializeCacheMap() 

if this is an MDL read 
call CcReadMdl() 

else 

call CcCopyRead(), using either an allocated MDL or the UserBuffer 

else (it's non-buffered) 

call LowLevelPostRequest() to send a Read request to the ECM 

if this is a synchronous, non-paging read 

update the current byte offset in the FileObject 
set the number of bytes read in the Status. Information field 
release any acquired resource and deallocate appropriate data structures 

Write 

Writes will definitely be open to async posting. A general algorithm: 
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if the write length is 0 

return success 
if the input file is a directory 

return error 
if the file not opened with write permissions 

return error 

if this IRP can be processed asynchronously 
post for async processing 

if this is a buffered write 

call CcCanlWrite() 

if false 

we have a hard error; fail 
if this is paging I/O 

acquire the PagingloResource shared 

else 

acquire the main Resource shared 
if the length is 

(Low == FILE_WRITE_TO_END_OFJ r ILE) && (High == Oxffffffff) 
we're to write at EOF 
if this is a non-paging, non-buffered write, and 

there is a mapped data section object for this file 
acquire the global EFS resource exclusive 
acquire the PagingloResource shared, starving exclusive waiters 
call CcCacheFlush() on the range for this write 
release the PagingloResource 
return error if the cache flush failed 

acquire and release the PagingloResource exclusive (to serialize) 
call CcPurgeCacheSection() on the range for this write 
release the global EFS resource 

if this is a paging I/O write 

if the starting offset is beyond the current EOF 

return success 
if the ending offset is beyond the current EOF 
truncate the write length to EOF 

if this is a buffered write 

if the private cache map is NULL 

call CclnitializeCacheMap() for this file 
if the write will extend the file size 

release the resource acquired 

re-acquire the resource exclusive 

call CcSetFileSizes() inform the cache manager 
if this is an MDL write 

call CcPrepareMdlWrite() 

else 

call CcCopyWriteQ, using either an allocated MDL or the UserBuffer 

else 

call LowLevelPostRequestQ to send the write request to the ECM 

set the number of bytes written in the Status. Information field 
update the fife size fields in the FCB if this write extends the length 
if this is a non-paging write 
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set the CCB modification flag 
if this write is synchronous 

update the CurrentByteOffset field in the FileObject 
release any acquired resource and deallocate appropriate data structures 

Fast I/O Read 

Initially at least, we'll just set the fast I/O read routine to FsRtlCopyRead(). 
Fast I/O Write 

Initially at least, we'll just set the fast I/O write routine to FsRtlCopyWrite(). 
Fast I/O Query Basic Info 

This will just fill in the basic info buffer with the data in the FCB. 
Fast I/O Query Standard Info 

This will just fill in the standard info buffer with the data in the FCB. 
Fast I/O Query Open 

This will just fill in the network open info buffer with the data in the FCB, if the file ex- 
ists. Some empirical observations I've made using NTFS: 

• Regardless of whether the file exists or not, this will return TRUE (all fast I/O 
routines are boolean) 

• If the file does not exist, it will set the EOF size in the buffer to 0. The Alloca- 
tionSize must be non-zero. All other fields seem to be don't cares. 

• If the file exists but is zero length, then both EOF and AllocationSize will be 0. 
. The IRP sent to this routine is for an IRP JvlJCREATE; we can use more than 

just the name to identify the file, but also the security characteristics or whatever 
else is sent in the IRP. 

File Query Info 

Standard queries will be supported; these however will not: 

• Filelnternallnformation— no OPEN_BY_FILE_ID 

• FileEalnformation — no EA data 

• FileCompressionlnformation — no on-disk compression 

• FileStreamlnformation — no multiple streams 



File Set Info 

These actions will be supported: 
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• EndOfFile size changes 

• AllocationSize changes 

• Time stamp changes 

• File position changes 

• File disposition changes— delete pending 

• File rename requests 

Here is a general algorithm: 

if AllocationSize or EOF size change 

if the new size is smaller than the current size 

call MmCanFileBeTruncated(), to ask the VMM if this is okay 
if yes 

call CcSetFileSizes() 
else 

return STATUSJJSERJWAPPED^FILE error 

else 

send a Metadata Set request to the ECM with new, extended size 
if status returned is error 

return error (disk space full) 
update AllocationSize and FileSize fields of required FCB header 

else if time stamp change 

send Metadata Set request to ECM 
if error returned 
return with error 

else if file position change 

update FileObject CurrentByteOffset field 

else if disposition (delete) change 
if the Delete flag in the IRP not set 

clear delete on close FCB flag 
if file already marked for delete on close 

return success 
if file not open with write permission 

return error 
if MmFlushlmageSectionQ fails 

return error 

if this is* a directory, and the directory is not empty 

return error 
set delete on close flag in FCB 

else if rename change 

if the source name is for a directory 

if the directory has any open files/directories under it 
return error 

if Parameters.SetFile.FileObject is NULL (simple rename) 
target dirname is the same as dimame of IRP FileObject 
target basename is Buffer.FileName 

else 

if Buffer.RootDirectory is NULL (fully-qualified rename) 



Omnishift Technologies, Inc. 



26 



Company Confidential 



eStream File System Driver Low Level Design 

fully qualified target name is Buffer.FileName 

else (relative rename) 

call ObReferenceObjectByHandle() to get the file object of the relative directory 

from file object, get (fully qualified) dirname of target 

append Buffer.Filename to root directory dirname to get fully qualified target 

if the target name isn't on EFS 

return error 
if target exists 

if Parameters.SetFile.ReplacelfExists is FALSE 

return error 
if target is a directory 

return error 
if target is read-only 

return error 
if target has any open handles to it 

return error 

call LowLevelPostRequest() to send a Rename request to the ECM 
Directory Query 

Directory queries turn into directory read requests to the ECM. The EFSD will take the 
contents of the read buffers and fill the IRP MN QUERY DIRECTORY buffers sent to 
it from the NT Executive by parsing the directory contents. 

Note: this design does not use the NT Cache Manager for metadata or for directory en- 
tries, nor does the EFSD store the directory contents anywhere in its data structures. It 
will always go back to the ECM for directory queries. Given that most directory queries 
occur in this order: 

Create 

Directory query 
Close 

unless we can cache the contents in a location more persistent than an FCB, we will need 
to resubmit the request to the ECM for each new directory query. If this poses a per- 
formance problem, we need to handle it then. 

Directory queries are subject to async processing. 

This is an ugly NT interface. Here are some general points regarding directory queries, 
and how they will be implemented on EFS: 

• These requests come in from the I/O Manager in a context-sensitive sequence. 
I.e., a request will come for the initial N directory entries; the next request will be 
for the next M entries; etc. Kind of like strtok(). 

• Thus, state must be maintained from request to request. This state will be kept in 
the CCB for a file stream, and consists of: 
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o Pattern sent in on first request 

o Index of n'th entry to start retrieving with 

• My experience is that the INDEX SPECIFIED flag is never set in a directory 
control query, even on queries subsequent to the initial one. 

• For EFSD, the Filelndex will represent the offset of the fixed-length portion of a 
directory entry as returned by the ECM. 

• Initially, at least, all directory queries will cause the EFSD to read all the entries 
for that directory from the ECM. We can modify this later if needed. 

Here is a general algorithm, based on the sources for the Fastfat driver: 

if the FileObject is not for a directory 
return error 

post this for async handling, to read all directory pages from the ECM 

if the CCB pattern field is empty and the CCB flag "match all" isn't set 
this is the initial query 
acquire the FCB Resource exclusive 

else 

acquire the FCB Resource shared 

get a pointer to the input buffer, using either an MDL or the UserBuffer 
if this is the initial query 

parse the FileName pattern 

save the pattern in the CCB 

if the pattern is u *" 

set the "match air flag in the CCB 

if SLRESTARTSCAN is specified 

use an index of 0 for the query 
else if SLJNDEX_SPECIFIED is specified 

use the input index for the query 

else 

if this is the initial query 
use an index of 0 

else 

use the index saved in the CCB 
start with the directory entry corresponding to the starting index 

if this index is beyond the number of entries in the directory, and this is not the initial query 

return STATUS_NO_MORE_FILES 
while there are more directory entries 

if the directory entry name matches the pattern in the CCB, 

using FsRtllslMamelnExpression() 
if there isn't room in the buffer for this entry 
break 

write the entry in the input buffer 

if there is a next directory entry beyond the current one 

the Filelndex field is set to the offset of this next directory entry 

else 

the Filelndex field is set to 0 
if there is a previously written entry 

fix up the NextEntryOffset in the previous entry to the byte offset of this entry 
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if SL RETURN SINGLE ENTRY specified 
break 

8-byte align the current pointer in the user buffer 
advance to next directory entry 

if we wrote nothing 

if we stopped because of no room 

return STATUS_BUFFER_OVERFLOW 

else 

return STATUS_NO_SUCH_FILE 

update the index field in the CCB 

Status.lnformation is set to the number of bytes written 

return STATUS_SUCCESS 

File System Query Info 

Empirically, IVe noticed that the LanMan redirector returns failure for most of these re- 
quests. So, except for any user-defined FSCTL requests we want to define, I'm going to 
fail all of these until it turns out we need to do otherwise. 

File System Set Info 

Ditto for this IRP type too. 
Volume Query Info 

We at least need to minimally implement these requests: 

• FileFsAttributelnformation 

• FileFsVolumelnformation 

• FileFsDevicelnformation 

This will be handled solely by the EFSD; the request will not go out to the ECM. File 
system size requests will not be handled. 

Volume Set Info 

We will fail all requests of this type. 
Flush Buffers 

A buffer flush request for a file stream will mean the following: 

• If the file stream isn't buffered, return immediately 

• The FCB main Resource is acquired exclusive 

. The Cache Manager is told to flush the buffer for the byte range of the file 

• The resource is released 
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A buffer flush for a directory is a successful NOP. 

The buffer flush request will insure that contents in the NT File Cache are written to the 
ECM (as a normal write request); the buffer flush request itself will not be propagated to 
the ECM. 

Testing design 
Unit testing plans 

The EFSD will be tested apart from integrating with the ECM or the rest of the eStream 
client. Some points: 

□ There will be a (relatively) simple stand-in user process for the ECM, to get re- 
quests from the EFSD and handle them locally. 

□ As much EFSD functionality as possible will be done using user-mode test cases 
(e.g., open files, read/write files, delete files, etc.). 

□ Some functionality may need to be unit tested using another kernel-mode driver to 
send explicit IRPs to the EFSD. 

□ Filemon will be used to monitor the requests being handled by the EFSD. 

Stress testing plans 

I've heard of a file system filter driver test package available from Microsoft. This is 
probably the best way we have of stress testing the EFSD. 

Coverage testing plans 

We'll try to measure coverage on the EFSD. If there is a general kernel-mode method for 
measuring coverage that's used company-wide, we'll exploit it. Otherwise, there will be 
some primitive self-coverage instrumentation conditionally inserted in the driver code 
that we can use at least for major code paths. 

Cross-component testing plans 

There's not much to do here apart from normal interactions with the ECM. 

Upgrading/Supportability/Deployment design 

For supportability, there will be solid debugging aids— using printfs— built into the 
EFSD sources. Additionally, aside from good error codes returned from its interfaces, 
the EFSD will explore diagnostic traces optionally dumped for deployment. 
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Issues with stakes in the ground 

□ At present, I am assuming that there is a single drive letter associated with the 
EFSD, though there's no technical reason why this must be so. If indeed we or- 
ganize the client system and the eStream file hierarchies to have multiple drive 
letters, either the EFSD or the ECM will need to parse the drive letter and do the 
right thing. 

□ This design assumes that 8.3 DOS-style filenames need not be supported. Adding 
such support will increase the complexity of the EFSD, as well as many other 
components in the eStream system: on the client, on various servers, and on the 
content builder. 

□ No support is provided in this design for: 

a. byte locks 

b. directory notification 

c. file open by file id 

d. file system control requests 

Open Issues 

1 . I'm unclear on how to use the NT File Cache for metadata and directory contents. 
For now, I'm ignoring such matters, and we will only be caching file contents. 

2. I do not know how to hook up the EFSD to UNC names. That is, I don't know 
how to set things up to have all file accesses like WASP l\Qffice\winword.exe be 
directed to the EFSD by the NT I/O Manager. 

3. This design doesn't cover exactly how IRPs are posted for asynchronous process- 
ing. The SFSD example in Rajeev Nagar's book really isn't sufficient for some 
of what we need to do. Also, it's unclear to me what value there is to returning 
STATUSPENDING and handling a request asynchronously if the caller is block- 
ing anyway. 
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Functionality 

The EStream Prefetching and Fetching [EPF] module requests eStream application pages. 
The module handles demand fetching & prefetching. Prefetching is based on information 
in the prefetch section of the AppInstallBlock, on spatial locality, and potentially on pro- 
filing data gathered during previous runs of the application on this client. 

Please see "eStream 1.0 Client Profile/Prefetch Straw Man" for background information 
related to prefetching & profiling. In particular, that document discusses the redundancy 
of gathering profiling information for prefetching on a client w/generous cache resources 
& the limitations of path-based prefetching when bandwidth is insufficient to hide latency 
without adequate execution-time lookahead. The collection & utilization of client profile 
information will be developed under an option (as it was for the prototype), and will be 
deployed in eStream 1.0 only if appropriate benefit is demonstrated. 

Data type definitions 

For core prefetching & fetching functionality, EPF uses the ECMRequest data structure. 

For initial prefetch data, EPF reads FileNum,PageNum pairs from a file given by AIM. 

For profiling data, EPF reads/writes FileNum,PageNum predecessor/successor per- 
AppID information from/to nonvolatile storage. This information is of a similar nature to 
the eStream cache and should be stored similarly on a per-client basis. For eStream 1 .0, 
profile data is only stored if user has indicated that eStream cache can grow w/o a bound. 

Interface definitions 
From AIM to EPF: 

• EPFPrefetchInitialAppBlocks(In AppID, In AppCorePrefetchSectionFileName, In 
AppOptionalPrefetchSectionFileName) 

From ECM to EPF: 

• EPFReportPage(In AppID, In FileNum, IN PageNum, In *Request) 

• EPFReadPage(In AppID, In FileNum, In PageNum, In *Request) 

From CUI to EPF: 

• EPFPrefetchSet(In OnOff) 
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From EPF: 

• LSMGetAccessToken(IN AppID, IN RequestHandle, IN ActivePrefetchOnly, 
IN RegisterCallback, OUT AccessTokenStatus, OUT AccessToken) 

• ECMIsPageInCache(In AppID, In FileNum, In PageNum, In Priority, Out 
PageState) 

• ECMReservePage(In AppID, In FileNum, In PageNum, In Priority) 

Component design 

Fetching and prefetching pages during program execution: 

Interface with ECM [run in an ECM thread] : 

ECM repeatedly calls EFSDGetRequest to pick up next eFSD request 
EFSDGetRequest allocates ECMRequest data struct X & initializes 
If the request is ERT_READ then 

ECM calls ECMReadPage w/ ECMRequest data struct X 

ECMReadPage acquires the work list/index lock 

If the READ hits in the cache then 

ECMReadPage calls EPFReportPage w/ Appld, FileNum, PageNum 

EPFReportPage allocates ECMRequest data structure Y, 

Puts the appropriate info including ERT_READ__HIT in it, 
Enqueues it in EPF's input queue & immediately returns 

ECMReadPage copies the data to the kernel and returns 

Else /* READ misses in the cache then */ 

ECMReadPage calls EPFReadPage with AppID, FileNum, PageNum 

EPFReadPage allocates ECMRequest data structure Y, 

Puts the appropriate info including ERT_READ in it, 
Enqueues it in EPF's input queue & immediately returns 

/* EFSDCompleteRequest is called when data comes back from Nw */ 

End If /* READ hit or miss in cache */ 

ECMReadPage releases the work list/index lock 
Else if request is ERT__WRITE then 

/* For prototype, only prefetched & profiled based on READs */ 
End if /* Request is ERT_READ or ERT_WRITE */ . 
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EPF processing of its input queue [run in EPF thread]: 

EPF repeatedly gets next ECMRequest in its input queue [using lock] : 

Switch (RequestType) 
Case ER THREAD : 

Calls ECMReservePage with ECM Request data struct Y 
/* Fall through */ 
Case ERT_READ_HIT: 

Call EPFRecordReference (In AppID, In FileNum, In PageNum) 
Call EPFPrefetchPredictedPath(In AppID, In FileNum, In PageNum) 
Break; 
End switch 

EPFPrefetchPredictedPath: 

EPF predicts path based on current reference, on look-ahead distance, 
& potentially on previous references including stored profile data 

For each page reference in the path 

EPF invokes ECMIsPagelnCache 

If the page is not either in the cache or in flight then 

EPF calls ECMReservePage w/null ECMRequest (designates prefetch) 
End if 
End for 

EPFRecordReference: 

EPF records pred/succ pairs of FileNum, PageNum on a per-AppID basis 

Prefetching pages specified in AppInstallBlock: 

At application install time, LSM calls ECMMount & then AIMInstall 

AIMInstall calls EPFPref etchlnitialAppBlocks which returns immedi- 
ately 

EPF uses separate prefetching thread to obtain this app's core pages: 

For each (FileNum, PageNum) pair from AppCorePref etchSectionFileName 

EPF invokes ECMReservePage w/priority parameter marked as criti- 
cal 

and w/null ECMRequest (designates prefetch) 

While cache utilization & client networking usage sufficiently low 

For each (FileNum, PageNum) from AppOptionalPref etchSectionFileName 
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EPF invokes ECMReservePage w/null ECMRequest (designates pre- 
fetch) 

Prefetching pages independent of execution: 

For client eStream installations with adequate caching resources, it may make sense to 
prefetch pages of files that are heavily-used & widely-covered. The winstone data indi- 
cated that such files include the main exe associated w/each application & certain core 
dlls. Hence, the idea of prefetching pages independent of eStream execution. This idea 
was not explored during the prototype phase & will be deployed in the eStream 1 .0 prod- 
uct if it proves itself. 

If the eStream client software is active for a particular user & it detects that the client sys- 
tem has been idle for an extended period of time, the independent prefetching system is 
engaged. The independent prefetching system decides which AppID would benefit from 
prefetching (somehow), which FileNum should have some of its pages prefetched (some- 
how), & invokes LSMGetAccessToken indicating ActivePrefetchOnly & specifying a 
callback routine to which LSM should report whether the AccessToken is obtained. Ac- 
tivePrefetchOnly implies that the AccessToken should not be renewed automatically (to 
facilitate pre-emption of the AccessToken by a client who wants to use the app) and it 
implies that any problems getting the AccessToken should result in giving up rather than 
launching some dialog with the user. If the AccessToken is obtained, then as long as 
there are adequate cache resources & as long as the client network usage is low (aside 
from the bandwidth being used for prefetching!), EPF fetches pages in FileNum that are 
missing from the cache. The independent prefetching system disengages whenever client 
load detected, whenever an AccessToken is denied, whenever the cache is enhanced or 
polluted enough, etc. 

Testing design 

Unit testing plans/Coverage testing plans 

A simple driver program, which will consist of a sequence of EPFReportPage & EP- 
FReadPage calls and stubs for ECMIsPagelnCache & ECMReservePage, will be used for 
unit testing of the runtime prefetching/fetching functionality. Expected profiling infor- 
mation will be compared with that collected and expected fetching/prefetching sequences 
will be compared with those produced. A dummy AppInstallBlock prefetch section file 
will be created and will drive unit testing of that portion of the prefetcher. 

Stress testing plans 

High fetching demand will be most stressful for this component. This may be tested by 
running many applications simultaneously on a client, forcing ECM to "miss" every ref- 
erence, but having CNI return the data fairly quickly to keep the pressure on EPF. 
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Cross-component testing plans 

This component is tightly coupled with the ECM. It will be integrated with the ECM as 
quickly as possible & will leverage off of ECM's testing infrastructure. 

Upgrading/Supportability/Deployment design 

A tracing feature will be introduced to have EPF report each fetch & prefetch it is making 
and why. An option to disable prefetching (and potentially profiling) will be provided. 
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Anne Holler * 




>* Version 1.0 



Introduction 

This document presents background issues related to the collection of eStream client 
profile data & its use in client prefetching and client cache replacement policy and lists 
proposed design decisions wrt client profiling and prefetching for eStream 1 .0. The 
document is intended to provide a baseline for discussions prior to proceeding with a 
detailed low level design. 



Client Profile Data Background 

eStream profile data may be used in client prefetching & client cache replacement policy. 
Ideally, eStream profile data consists of a sequence of (FileNum,PageNum) entries which 
comprise all references made on behalf of some run(s) of a particular eStream app, with 
the idea that any of these references would potentially result in an eFS request. Dynamic 
page tracing based on page faults gives a subset of the ideal trace, which includes the 
page references for compulsory misses, some subset [possibly empty] of the page 
references for the non-compulsory misses, and no entries for the remaining page 
references. A recently proposed augmentation of this data with information from static 
analysis of executables misses indirect links between pages and is in general imperfect; 
subprogram entry points will not always be visible and code & data may be interleaved 
on the same page in x86 code. Gathering ideal eStream profile data is a heavy-weight 
process, involving instruction level tracing. 

What kinds of issues could arise with respect to incomplete eStream profile data? To 
illustrate, let us assume that the sequence {b,c,d,x,y,z} appears in an incomplete eStream 
profile as {b,z} . One issue is that we do not know how far ahead a page in an incomplete 
profile will actually be used; prefetching z at the time b is being accessed is a problem if 
z displaces any of {c,d,x,y}. Another issue is that we may miss prefetch opportunities; 
the incomplete profile does not indicate that we need to prefetch {c,d,x,y} if accesses to 
them page fault and the pages are not in our cache. Another issue is that incomplete 
profile data does not allow us to know which pages are used frequently; if b is used 
frequently, we might see only one reference to it per run in the ordered profile data 
(because its frame tends to stay in main memory), whereas if z is used infrequently, we 
might see no references to it on some runs and possibly multiple references to it on other 
runs (assuming its infrequent use means its frame tends to get displaced from physical 
memory). Hence, straightforward use of either reference counts or LRU based on 
incomplete eStream profile data can be flawed wrt client cache replacement. 
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If incomplete profile data based on page faults is both collected and used on the same 
client, the issues cited above are less likely to occur (assuming the execution load on the 
client is roughly the same each time the application is invoked); profile data collected on 
an eStream builder machine and downloaded to a customer client machine at install time 
would be expected to be more likely to encounter these issues. While various approaches 
have been discussed to mitigate the potential impact of these problems, the decision has 
been made to forgo the profile section of the AppInstallBlock for eStream 1 .0 (largely 
because the planned lightweight builder training strategy means that the profile section 
contents would be redundant with the prefetch section contents). Uploading client profile 
data, which has previously been discussed, is also outplan for eStream 1 .0; since there is 
no initial downloaded profile data, there is no need to use uploaded client profile data to 
refine it. Application usage data, which may be of interest to ISVs and others, will be 
gathered at a less granular level from the SLM server. 

What good is page- fault-based client profile data collected & used on the same client? If 
the client has a cache of unlimited size, in which no block is ever replaced, this profile 
data is redundant in the sense that it brings no obvious value to eStream 1 .0 in addition to 
what is already provided by the contents of the client cache. If the client cache does need 
to do replacement, this profile data can be used to guide the replacement strategy, but 
only if the data is used in a clever way to avoid the oddities that result from its 
incompleteness (e.g., use per run); if it is not used cleverly, one may as well use random 
replacement. Client cache replacement implies that refetching of displaced pages may 
be needed. The profile data can potentially do a good job of supporting prefetching of 
this refetching; in experiments with the prototype, one can delete a warmed client 
eStream cache, just use the profile data, & hide virtually all latency associated with the 
refetches, at least in the scenario in which there is adequate bandwidth. For a 100Mbps 
link, executing start/exit of word, cache hit rate with profile-guided prefetching is 98%, 
with 6% additional overhead for redundant prefetch. 

How does profile-based prefetching compare with simple locality-based prefetching, 
for which no special technology is needed to collect and use the profiles is needed? In 
experiments with the prototype, locality-based prefetching yielded not-too-terrible results 
as compared with profile-based prefetching wrt client cache hit rate, but at the cost of a 
much higher wasted prefetch rate, i.e., at the cost of much more network bandwidth and 
server traffic. For 100Mbps link, executing start/exit of word, cache hit rate with 
locality-guided prefetching is 75%, with 89% additional overhead from redundant 
prefetch. We need to decide if client profiling is worthwhile for eStream 1.0, based on 
whether we believe there will be sufficient cache replacement and if so, whether we want 
to avoid the extra overhead of locality prefetching to restore displaced pages and if there 
will be sufficient bandwidth to exploit the extra knowledge that profile data gives us 
[perhaps via idle prefetch]. Putting a stake in the ground, I advocate designing and 
building client profile gathering and associated prefetching, because I believe it may be 
useful and because I believe we will be able to leam things from it which may profit us in 
the future. Please note that for the prototype, client profile gathering was under an #ifdef, 
so it could be removed as desired, since it adds memory consumption and overhead to the 
eStream client. We could develop this way for the real product as well, in case we decide 
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that it is not worthwhile for the eStream 1.0 product at all or at least for some particular 
variant of the product which is geared to very fat cache clients. 

Please note that, in general, client profiling is done on a per application basis; if the user 
is running several eStream applications on his client, it is expected that we would want to 
learn page sequences within each application, not sequence relationships across apps. 
Similarly, client profiling is done on a per invocation basis; if the user is running several 
instances of a particular eStream application, it is expected that we would want to learn 
about the sequences belonging to each instance separately. For the prototype, we turned 
off learning profile sequences if more than one instance of an application was running; 
we may want to consider doing this for the real product as well. 



Client Prefetching Background 

When does a client do eStream prefetching? The most intuitive time to do eStream client 
prefetching is at eStream app runtime, in association with runtime behavior; this is the 
only time at which prefetching is done in the prototype. Two other times to consider 
doing eStream prefetching are at app installation time and at network idle time. At app . 
installation time, the AppInstallBlock contains a prefetch section, which specifies 
{FileNum,PageNum} pairs for pages which are recommended to prewarm the client 
cache for this app; the client prefetcher will ask the client networking code to request 
these pages from the appropriate app server. At network idle time (i.e., when there is 
little client networking traffic), app pages may be downloaded without directly impacting 
client eStream performance; clearly such prefetching would need to be done judiciously 
to avoid polluting the client's cache, impacting server scaling, wasting network 
bandwidth, or hogging the app's license, but this could be a interesting strategy for 
eStream technology if it were done effectively. 

What does the eStream client prefetch? As discussed in the previous section on client 
profiling, profiling info can be used to drive which pages to prefetch, either by 
prefetching along an observed path or by prefetching according to a projected pattern, 
possibly based on observed references. Spatial locality is a frequently observed pattern in 
general, and may be used as the default projected pattern in the absence of relevant 
profiling information. In eStream 1.0, we may also consider using semantic information . 
to drive prefetching. One example might be prefetching the contents of the directory files 
for an installed app, since directory files may contain useful FileEDs and metadata. 
Another example might be to prefetch any missing pages of heavily used files such as the 
main executable for each application, with the idea that the pages might eventually be 
used for some app feature not yet exercised. Several folks have suggested that we 
gradually download the entire app, if cache space allows, with the vision that this 
provides full native performance; if this can be done without impacting any of our core 
selling points, this might be worth considering. 
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Client Profile Data Collection Design Proposal 

Here are key points on the design of eStream 1 .0 client profile data collection: 

*) Have profiler see all requests coming from eFSD 
*) Record profile data on per-application basis 

*) Record profile sequence for app only if single instance of app running (how can tell?) 
*) Represent (potentially lengthy) profile data list (with some loss) as (pred,succ) tuples 
*) Make client profiler optional 

*) Mark use-per-run pages high priority to aid cache replacement policy 



Client Prefetching Design Proposal 

Here are key points on the design of eStream L0 client prefetching: 

*) Prefetch at app install, mark specified pages as high priority to avoid their replacement 
*) Also prefetch at app runtime, have prefetcher see all requests coming from eFSD 
*) If eFSD page request on known path, project next refs on path; else, use spatial locality 
*) Prefetch only projected refs that are not already in client cache 

*) Consider network speed, path likelihood, cache issues in determining prefetch distance 
*) Also prefetch at idle, if unlimited cache; get missing pages from app's main executable 
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eStream File Spoofer Low Level Design 



Curt Wohlgemuth 
Version 2. 0 




Functionality 

The eStream file spoofer is a kernel-mode driver responsible for redirecting file accesses 
from local file systems to the eStream file system driver. It is implemented as a file sys- 
tem filter driver that traps all IRP requests to the file system device handling drives that 
must be spoofed, and redirects these requests to the EFSD. 

Data type definitions 

The file spoofer will understand entries in the "file spoof database" as they have been 
identified by the eStream builder and installed by the app install manager, but these are 
not defined by this component. 

Entries in this spoof database will have the following entries: 

□ original (fully qualified) path name of file: this resides somewhere on a local disk 
of the client machine 

o new (fully qualified) path name of the file to spoof to: this resides on the eStream 
file system drive 

The spoof database will reside in the registry, so it can be persistent across reboots, and 
so the file spoofer need not open a file to load it. As applications are installed on a client 
machine, the Application Install Manager will load new spoof entries into the registry, 
and inform the file spoofer that it must reload this database. Similarly, when an app is 
uninstalled, the AIM will remove spoof entries from the registry, and inform the file 
spoofer. 

This proposes that the each spoof entry is a separate name/value entry under a single key 
in the registry: 

□ Name: the original filename 

□ Value: the new filename 

Interface definitions 

The eStream file spoofer is called by two components: 

1 . The eStream client start service, which will start and stop the file spoofer 
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2. The AIM, which will inform the file spoofer to reload the spoof database from the 
registry 

The interfaces called by both of these user-mode components will be in the form of De- 
viceIoControl() calls. The following IOCTL codes will be defined for use by callers of 
the file spoofer: 

IOCTL_FS_START_SPOOFING 
I OCTL_FS_STO P_S POOF I NG 
I OCTL_FS_RELOAD_S POOF_DB 

Starting spoofing 

The input buffer for this IOCTL should supply the name of the registry key containing 
the spoof entries as values. The output buffer for this IOCTL is ignored and should be 
NULL. 

This will return either STATUSJSUCCESS, or an error return status if something goes 
wrong. It causes the file spoofer to read the spoof registry entries, and load up each entry 
into memory. 

Note that starting spoofing is currently identical to reloading the spoof database. 
This is called by the eStream client start service on startup. 

Stopping spoofing 

The input and output buffers for this IOCTL are ignored and should be NULL. This will 
return either STATUSJSUCCESS, or an error return status if something goes wrong. It 
causes the file spoofer to clear its memory image of the spoof database. 

This is called by the eStream client start service on shutdown. 

Reload spoof database 

The input buffer for this IOCTL should supply the name of the registry key containing 
the spoof entries as values. The output buffer for this IOCTL is ignored and should be 
NULL. 

This will return either STATUSJSUCCESS, or an error return status if something goes 
wrong. It causes the file spoofer to read the spoof registry entries, and load up each entry 
into memory. 

Note that this is currently identical to starting the spoof database. 
This is called by the AIM when a new eStream app is installed. 

Omnishift Technologies, Inc. 2 Company Confidential 



eStream File Spoofer Low Level Design 

Component design 

The file spoofer will have these major tasks: 

□ Track the following data: 

o all current valid file spoof entries 
o spoof entries by filtered file system 

□ Filter native file system drivers for local drives, intercept all IRPJVIJ CREATE 
and FAST_IO_QUERY_OPEN requests, and for spoofed files, change the file- 
name of the FileObject associated with these requests. 

Data structures 

The file spoofer needs to be able to quickly look up a filename in the in-memory spoof 
database. The current design will use a hash table, whose size and hash function will be 
tuned as we get experience with real applications. 

Adding or deleting entries from the hash table will by synchronized using a global re- 
source. 

Algorithms 

Here are basic algorithms for these steps. 
Load spoof database 

This reads all the name/value pairs under the registry key which holds the spoof entries, 
loads them into a temporary hash table, then points the real hash table to this one. 

traverse the registry until the input registry key is opened, using ZwOpenKeyO 
if not found 

return error 
if no name/value pairs exist in this key 

return "no data" 

for each name/value pair found with ZwEnumerateValueKeyO 
build a hash node for this and insert it into temp hash table 
if the drive letter for the old filename entry is one we're not currently filtering 
put this drive letter on new drive list 

acquire the hash table resource exclusively 

point the global hash table head pointer to the temp hash table 

release the resource 

for each drive letter on the new drive list 
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look up FS device for this drive 
if we really aren't attached to it 

attach self to this FS device as filter driver 

free the old hash table 
free the drive list 
return success 

Stop spoofing 

acquire hash table resource exclusively 
free the global hash table 
detach self from all filtered FS devices 
release hash table resource 

Trap Create and QueryOpen requests 

acquire the hash table resource shared 
if hash table head pointer is non-NULL 

look up input filename in hash table 
release hash table resource 
if filename not found in hash table 

send I/O request to original target FS driver 
else 

free memory of existing file name in input FileObject 

allocate memory for new, spoofed filename, copy into this memory 

send I/O request to eStream file system driver 

Testing design 
Unit testing plans 

The file spoofer will be tested as a standalone component, apart from the rest of the 
eStream client. A driver test program will be written to test all functionality and corner 
cases. This includes filtering all FSDs active for a client system, and multiple drives 
handled by a single FSD. 

Stress testing plans 

The file spoofer should be able to work, with little or no performance cost to the system 
as a whole, even when the attached FSDs are under heavy load. Some stress testing will 
be done in this fashion. 
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Coverage testing plans 

If we come up with a method for measuring coverage for kernel-mode components, we'll 
do so for the file spoofer as well. 

Cross-component testing plans 

Not clear if anything need be done here outside of the standard execution of the eStream 
client. 

Upgrading/Supportability/Deployment design 

I don't see any upgrade/compatibility issues for the file spoofer. For supportability, there 
will be a good debugging strategy and sufficient error message return codes for the caller. 

Open Issues 

• This is a list of issues that need to be further investigated or revisited during implementa- 
tion. 

1 . We will need to experiment with the hash table to tune it for fast lookup. It's pos- 
sible that we may need to replace the hash table with a faster lookup algorithm. 
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Functionality 

This document presents the low level design of the License Subscription Manager 
[LSM], an eStream client module that tracks information about ASP accounts, applica- 
tion subscriptions, and installed applications. The document also describes the interface 
to the eStream Client Browser Module [CBM], a client component that allows an ASP 
web server to notify LSM of client-relevant changes in an ASP account. (The CBM in- 
terface is used in specific situations, as described below; ASP account changes are de- 
tected automatically in steady-state operation.) 

Data definitions 



There are two data sets that LSM maintains on a per-user basis on the client [italics => 
only volatile storage]. One set is associated with the ASPs from which the client 
eStreams applications & the other is associated with the eStream applications to which 
the client is subscribed. This data is kept in the registry. 



ASP Data Set 



• ulntl28 AspID - Local handle for client convenience 

• string UserName 

• string Password - Only volatile storage unless user approves nonvolatile 

• string ASPWebServerName - May simply be an ip address 

• eStreamServerSet SLiMServerSet 

Subscription Data Set 



• ulnt 1 28 Subscription^ 

• ulntl28 AppID 

• Int32 RootFileNumber 

• eStreamServerSet AppServerSet 

• eStreamAccessToken AccessToken 

• ulnt64 ExpectedExpirationTime 

• enum {Installed, NotlnstalledPrompt, NotlnstalledDontPrompt) AppListallStatus 

• enum {Updated,NotUpdatedPrompt,NotUpdatedDontPrompt} AppUpgradeStatus 

• enum {ActivelnUse, ActivePrefetchOnly, Inactive} AppActiveStatus 

• string AppName 

• string VersionName 

• string Message 

• ulntl28 AspID - Local handle 
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Interface definitions 

Client Component Acronyms 

• AIM: Application Installation Manager 

• CBM: Client Browser Module 

• CES: Client EStream Startup 

• CNI: Client Network Interface 

• CUI: Client User Interface 

• ECM: EStream Cache Manager 

• LSM: License Subscription Manager 



LSM InterfacesTo/From Server Components 
From LSM to SLiMServer via CNI: 

• CNIGetSubscriptionList(IN AspID, IN UserName, IN Password) 

• CNIGetLatestApplicationInfo(IN AspE), IN SubscriptionED) 

• CNIAcquireAccessToken(IN AspID, IN SubscriptionID, IN UserName, IN 
Password) 

• CNIRenewAccessToken(IN AspID, IN SubscriptionID, IN UserName, IN Pass- 
word, IN AccessToken) 

• CNIReleaseAccessToken(IN AspID, IN SubscriptionID, IN UserName, IN 
Password, IN AccessToken) 

• CNIRefreshAppServerSet(IN AspID, IN SubscriptionID IN AccessToken, IN 
BadQos, IN NoService) 

From ASPWebServer to CBM: 

• ASPNotify(IN UserName, IN SLiMServerSet) 
From LSM to ASPWebServer: 

• PromptASPNotify: HTTP POST to server cgi script to engender an ASPNotify 
LSM Interfaces To/From Client Components 

From ECM to LSM: 

• LSMGetAccessToken(ESf AppID, IN RequestHandle, IN ActivelnUse, IN Regis- 
terCallback, OUT AccessTokenStatus, OUT AccessToken) 

* LSMReleaseToken(IN AppID) 
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From LSM to ECM: 

• ECMMount(IN AppID, IN RootFileNumber) 

• ECMRemount(IN AppID, IN OldRootFileNumber, IN NewRootFileNumber) 

• ECMSetTokenState(IN AppID, IN State) - NotHolding, Holding, Denied 

From CBM to LSM: 

• LSMAspAccntPing(IN AspWebServerName, IN UserName, IN SLiMServerSet) 

From CES to LSM: 

• LSMInitialize(IN WindowsUserName) -- Read ASP & Subscription Data Sets 
from registry 

• LSMUpdateAHSubscriptionStatus(void) 

From LSM to AIM: 

• AIMInstall(IN AppED, IN AppInstallBlockFileName, OUT InstallStatus) 

• AIMUninstall(IN AppID, OUT UninstallStatus) 

From LSM to CUI: 

• CUIInformUser(IN Message) 

• CUIAskUserYesNo(IN Message, IN CheckAskAgain, OUT Response, OUT 
DontAskAgain) 

• CUIAskUserPassword(IN Message, OUT Response, OUT Retain) 

From CUI to LSM: 

• LSMUpdateAHSubscriptionStatus(void) 

• LSMGetAspList(OUT NumAsps, OUT AspID[]) 

• LSMGetAspInfo(IN AspID, OUT ASPWebServerName, OUT UserName) 

• LSMGetAppList(IN AspID, OUT NumApps, OUT AppID[]) 

• LSMGetAppInfo(IN AppID, OUT AppName, OUT VersionName, OUT Mes- 
sage, OUT AppInstallStatus, OUT AppUpdateStatus) 

• LSMInstall(IN AppID, OUT InstallStatus) 

• LSMUninstaII(IN AppID, OUT UninstallStatus) 

• LSMUpgrade(TN AppID, OUT UpgradeStatus) 

From EPF to LSM: 

• LSMGetAccessToken(IN AppID, IN RequestHandle, IN ActivePrefetchOnly, 
IN RegisterCallback, OUT AccessTokenStatus, OUT AccessToken) 
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From CNI to LSM: 

• LSMGetAppServerSet(IN AppID, OUT AppServerSet) 

• LSMGetNewAppServerSet(IN AppID, OUT AppServerSet) 

• LSMGetSLiMServerSet(IN Asp ID, OUT SLiMServerSet) 

• LSMGetNewSLiMServerSet(IN AspID, OUT SLiMServerSet) 

• LSMGetAccessToken(IN AppID, IN RequestHandle, IN Unknown, IN Regis- 
terCallback, OUT AccessTokenStatus, OUT AccessToken) 

• LSMReturnSubscriptionList(IN AspID, IN ReturnCodes, IN NumberOfSub- 
scriptions, IN SubscriptionID[]) 

• LSMReturnLatestApplicationInfo(IN AspID, IN SubscriptionID, IN Return- 
Codes, IN Upgradelnfo) 

• LSMReturnAcquireAccessToken(IN AspID, IN SubscriptionID, IN Return- 
Codes, IN AccessToken, IN RenewalFreq, IN AppServerSet, IN Upgradelnfo) 

• LSMReturnRenewAccessToken(IN AspID, IN SubscriptionID, IN Return- 
Codes, IN AccessToken, IN RenewalFreq, IN AppServerSet) 

• LSMReturnReIeaseAccessToken(IN AspID, IN SubscriptionID, IN Return- 
Codes) 

• LSMReturnRefreshAppServerSet(IN AspID, IN SubscriptionID, IN Return- 
Codes, IN ServerSet) . 

Component design 

The LSM is a service module that handles requests from eStream client software and 
from the ASP Webserver via the CBM [discussed under its own heading at the end of this 
section]. Hence, LSM's component design is presented here in terms of how it responds 
to the requests that it receives in the three main scenarios in which it participates: running 
an eStream application, getting/refreshing application subscription information, and dis- 
playing subscription information. 

The web pages at \\fserv2\home\webpages\pmain.html are intended to provide a frame- 
work for discussing overall issues related to subscribe/install/uninstall scenarios. The 
discussion of scenarios in this section is at a much lower level than that material. 

Run an Application 

• ECM invokes LSMGetAccessToken whenever there is a reference to a file be- 
longing to a particular AppID for which ECM's internal data structure indicates 
that the AccessToken is in the NotHolding state. [ECM can determine the AppID 
of an eStream app by inspecting the fully-specified eFSD filename; the first level 
of the directory path is a string version of the AppID. If LSM does not recognize 
this AppID for some reason, an appropriate status is returned to ECM.] 

• LSM marks the AppID as ActivelnUse in the Subscription Data Set & returns the 
appropriate status to ECM. If the status indicates that LSM has an AccessToken 
for the application, ECM updates its internal data structure to indicate the Ac- 
cessToken is in the Holding state & continues. If the status indicates that LSM 
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does not have an AccessToken for the application, LSM takes ownership of the 
specified request, holding the RequestHandle in an internal queue for an eventual 
reissue request to eFSD. 

• LSM issues a CNIAcquireAccessToken call via CNI to a SLiMServer & when 
LSM receives the AccessToken via the LSMReturn Acquire AccessToken call- 
back, it places a copy of it in the volatile Subscription Data Set, assigning Expect- 
edExpirationTime according to the RenewalFrequency indicated by the SLiM- 
Server and setting up a timer alarm to go off shortly before the AccessToken is 
expected to expire. It calls ECMSetTokenState to notify ECM that the Access- 
Token is in the Holding state for the AppID. 

• LSM compares the application's current RootFileNumber on the client with Up- 
gradelnfo.RootFileNumber returned wrt the CNIAcquireAccessToken call; if the 
latter is higher & Upgradelnfo.ForcedUpgrade is off, it calls CUIAskUserYesNo 
about upgrading. LSM sets AppUpgradeStatus and RootFileNumber fields in the 
Subscription Data Set appropriately & calls ECMRemount if there is a minor 
upgrade. 

• Should an AccessToken be denied due to a mandatory major upgrade that costs 
money, LSM calls CUIInformUser to let him know that he needs to interact with 
his ASP to revise his account. 

• Should an AccessToken be denied due to a mandatory major upgrade that does 
not cost money, LSM calls CUIAskUserYesNo about upgrading. If he responds 
affirmatively, the Subscription Data Set is updated, AIMUninstall is called to 
nuke the current version, & AIMInstall is called to install the upgraded version. 

• Should an AccessToken be denied because it is currently held by another of the 
user's clients, a copy of the other client's AccessToken is returned. If the user re- 
sponds affirmatively to CUIAskUserYesNo("Yank?"), then LSM will call CUI- 
InformUser("Please wait"), send the token to CNIReleaseAccessToken, & then 
retry CNIAcquireAccessToken at the appropriate time. 

• For AccessToken denials that stick [either because the user has decided not to 
take a mandatory upgrade at this time, or because the user has decided not to yank 
another of his clients' AccessToken, or because the user has some problem with 
his ASP account (didn't pay, account suspended/terminated, got evicted, authori- 
zation failed)], the user is informed of the problem and told to save/exit the appli- 
cation ASAP. ECMSetTokenState is called to mark the token state Denied and 
EFSD is told to retry the request. When ECM's internal data structure has the 
AccessToken status for an application marked Denied, it allows the user to con- 
tinue running out of the cache, but will report failure to EFSD immediately on any 
access that misses in the cache. 

• CNI frequently invokes LSMGetSLiMServerSet to obtain SLiMServers for Ac- 
cessToken requests. Should CNI have quality of service issues with those servers, 
it calls LSMGetNewSLiMServerSet; LSM uses the PromptASPNotify interface 
to engender the ASP Web Server to send a message containing a SLiMServerSet. 

• Whenever an AccessToken timer alarm goes off, LSM checks that the associated 
app is marked ActivelnUse & if so, it issues a CNIRenew AccessToken & puts 
token returned by LSMReturnRenewAccessToken in the Subscription Data Set, 
resetting ExpectedExpirationTime and scheduling another timer alarm. 
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• When CNI needs to fetch requested data from an AppServer, it invokes LSMGe- 
tAccessToken. 

• CNI frequently calls LSMGetAppServerSet to obtain a set of AppServers 
through which to cycle; LSM returns the servers associated with the most recently 
acquired AccessToken. Should CNI be dissatisfied with the quality of service as- 
sociated with the AppServers it is given, it calls LSMGetNewAppServerSet 
which sends CNIRefreshAppServerSet to a SliMServer and receives a LSMRe- 
turnRefreshAppServerSet callback. 

• Should a CNI request be denied by an AppServer because the submitted Access- 
Token was expired, CNI calls LSMGetAccessToken, which operates as de- 
scribed above with respect to acquiring the AccessToken and with respect to 
sending a reissue request to eFSD. 

• ECM invokes LSMReleaseToken whenever there is a transition from 1 to 0 open 
files wrt a particular eStream app, setting the AccessToken status in its internal 
data structure to NotHolding (from either Holding or Denied). LSM sends a 
CNIReleaseAccessToken call to a SLiMServer [possibly only if the Expect- 
edExpirationTime for the AppID in question is not in the past] & receives an 
LSMReturnReleaseAccessToken callback. LSM marks the application as Inac- 
tive in the Subscription Data Set, indicating that automatic AccessToken renewal 
should not be continued for this application [LSM may also cancel any out- 
standing alarm]. 

Get/Refresh Subscription Information 

• Whenever the CBM receives an ASPNotify from an ASPWebServer, it sends an 
LSMAspAccntPing message to the LSM. The LSM checks its ASP Data Set to 
see if the specified ASP is one that it recognizes. If LSM does not recognize the 
ASP, it allocates a new ASP Data Set entry and calls CUIAskUserPassword to 
prompt the user for the password associated with this ASP+UserName (asking the 
user if the password can be saved on the client for future use). If LSM does rec- 
ognize the ASP, it only calls CUIAskUserPassword if the user did not allow the 
password to be previously recorded. 

• If the user indicates that his password can be saved on the client for future use, we 
encrypt the password using a reasonably secure symmetric algorithm, such as 
DES, the key being a hash of the AspID and the username. If we implement the 
eStreamClientExecutable as a service (outstanding implementation issue in CUI 
LLD), we store the password in a registry key that cannot be read by ordinary user 
accounts (services can run in the context of a separate account), else we store the 
password in a registry key belonging to the user. 

• Once LSM has the UserName & Password, it issues a CNIGetSubscriptionList 
request to one of the ASP's SLiMServers. For any SubscriptionlDs associated 
with that ASP that are installed on the client but do not appear in the list returned 
by the LSMReturnSubscriptionList callback, the user is informed that he cannot 
run the app at the current time & is asked if he would like it uninstalled from his 
client (note that we may be legally required to remove it from the cache, so this 
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may not be a question but rather just a message); AIMUninstall is invoked to 
achieve this. 

• For all other SubscriptionlDs, a CNIGetLatestApplicationlnfo request is sent to 
a SliMServer & a LSMReturnLatestAppIicationlnfo callback provides the data. 

• For SubscriptionlDs already in the Subscription Data Set, the RootFileNumber is 
compared to see if the application has been upgraded. If it has, the user is in- 
formed; the new RootFileNumber is recorded (assuming either that the upgrade is 
mandatory or that the user indicated he wanted it) and ECMRemount is called to 
notify ECM about the upgrade so that it can take appropriate action. [Please note 
that for eStream 1 .0, we will not support minor upgrades that involve new Appln- 
stallBlocks; if a new AppInstallBlock is needed, the upgrade is by definition a 
major upgrade & the app needs to have a new AppID assigned & uninstall/install 
actions taken.] 

• For SubscriptionlDs that were not already in the Subscription Data Set, a new 
Subscription Data Set entry is created and CUIAskUserYesNo("Install App?") is 
invoked. If user responds in the affirmative, LSM requests an AccessToken, calls 
ECMMount to notify ECM to prepare to service requests on behalf of this new 
application & then invokes AIMInstaH, which can use the eFSD/ECM path to 
obtain the AppInstallBlock. If user responds in the negative, he is able to indicate 
"don't ask again'Y"ask again later", which is stored in the Subscription Data Set. 

• Whenever the eStream client software starts up or at specific times requested by 
the user via the ClientUI, LSMUpdateAlISubscriptionStatus is called to refresh 
the eStream client's knowledge of the apps to which the user is subscribed. 

Display Subscription Information 

• The CUI provides a user interface to display ASP and application info, using 
LSMGetAspList, LSMGetAspInfo, LSMGetAppList, LSMGetAppInfo. 

• Please see "eStream 1 .0 Client User Interface Low Level Design" for information 
on LSMInstall, LSMUnin stall, and LSMUpgrade. 

Client Browser Module Overview 

The CBM is an eStream client component that runs in association with the client browser. 
CBM communicates a small amount of data to the core eStream client software (as de- 
scribed above) and it is realizable on both Internet Explorer 4+ and Netscape Navigator 
4+. It does not have to start the eStreamClientExecutable if it is not already running. 

Testing design 

Unit testing plans/Coverage testing plans 

For unit testing of LSM, a simple driver program will be written which will simulate the 
three main scenarios highlighted in this design document. It will exercise all LSM exter- 
nal entry points (listed in bold in the Interface Definitions section of this document) and 
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will achieve -100% PFA coverage of all non-assert paths as measured by Rational Pure- 
Cov. A process will be introduced to automate the execution of this testing. 

Cross-component testing plans 

As defined in the high-level design, LSM has many cross-component interfaces. In the 
steady-state operation of eStream, its most heavily-used interfaces will be those with 
ECM and CNI. It is anticipated that LSM will be integrated first with these two pieces 
and will be added to an automated nightly testing of running eStream applications. 

At eStream client software activation time, the CES interface is exercised; integration 
with CES and automated testing of the pair should be deployed as soon as both are avail- 
able. At application install and uninstall time, the AIM interface is utilized; presumably, 
automated testing of installation & uninstallation of apps will be key to the AIM testing 
strategy and can be combined with the testing of this module when the two are integrated. 

The CUI has a number of interfaces to LSM. Automated testing of the pair will be based 
on a UI testing tool like Rational's Visual Test tool (which BTW drives Winstone). 

Stress testing plans 

The most stressful usage scenario for LSM is high client load, i.e., many eStream applica- 
tions running simultaneously. Visual Test will be used to set up this kind of testing sce- 
nario. 

Upgrading/Supportability/Deployment design 

LSM is heavily involved in user scenarios involving upgrading, supporting, and deploy- 
ing eStream applications. The normal operation of these scenarios is automatic and 
seamless, but all are intended to have CUI interfaces that will allow manual intervention 
and troubleshooting. 

In addition, LSM will provide a debugging interface for dumping the in-memory images 
of the ASP Data Set and the Subscription Data Set. 

Implementation Issues 

• Develop mechanism for implementing CBM. The plan is to make CBM a browser 
helper application, i.e., a standard windows32 app that is registered to handle a 
particular MIME encoded filetype (*.estream) in the HTML stream. [BTW, this 
is done by Software Wow.] As an alternative proposal, Bhaven created JavaScript 
that starts Excel & communicates with it, though this solution seems to require 
that the browser be Internet Explorer, that the app be ActiveX, and that the secu- 
rity setting on the browser be at minimum setting. 
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• LSM may move to kernel space and its implementation will avoid constructs 
which might unduly complicate this potential migration. 
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Introduction 

This document presents background information related to the License Subscription 
Manager [LSM], an eStream client module that tracks information about ASP accounts, 
application subscriptions, and installed applications. The document also describes the 
interface to the eStream Client Browser Module [CBM], a client component that allows 
an ASP web server to notify LSM of client-relevant changes in an ASP account. (The 
CBM interface is used in specific situations, as described below; ASP account changes 
are detected automatically in steady-state operation). The document is intended to 
provide a baseline for discussions prior to proceeding with a detailed low-level design. 

LSM Data Sets 

There are two data sets that LSM maintains on the client in nonvolatile memory. One set 
is associated with the ASPs from which the client eStreams applications and the other is 
associated with the eStream applications to which the client is subscribed. 

ASP Data Set 

ulntl28 AspID — Local handle for client convenience 

string UserName 

string Password 

string ASPWebServerName 

eStreamServerSet SLiMServerSet 

Subscription Data Set 

ulntl28 SubscriptionID 

ulntl28 AppID 

Int32 RootFileNumber 

eStreamServerSet AppServerSet 

eStreamAccessToken AccessToken 

ulnt64 ExpectedExpirationTime 

enum {Installed, NotlnstalledPrompt, NotlnstalledDontPrompt} AppInstallStatus 
enum {ActivelnUse, ActivePrefetchOnly, Inactive} AppActiveStatus 
string AppName 
string VersionName 
string Message 

ulntl28 AspED - Local handle 
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LSM Interfaces To/From Client Components 



The use of the interfaces listed below is discussed in the LSM Functionality Overview 
section of this document; the interfaces are summarized here for easy access. 

Client Component Acronyms 
AIM: Application Installation Manager 
CBM: Client Browser Module 
CES: Client EStream Startup 
CNI: Client Network Interface 
CUI: Client User Interface 
ECM: EStream Cache Manager 
LSM: License Subscription Manager 

From ECM to LSM: 
LSMNotifyAppStart(IN AppID, OUT AppStatus, OUT RootFileNumber) 
LSMNotifyAppStop(IN AppID) 

LSMCheckAccessToken(IN AppID, OUT AccessTokenStatus) 
LSMGetAccessToken(IN AppID, OUT AccessToken) 

From LSM to ECM: 
ECMMount(IN AppID, IN RootFileNumber) 

From CBM to LSM: 
LSMAspAccountPing(IN AspWebServerName, IN UserName, IN SLiMServerSet) 

From CES to LSM: 
LSMUpdateAllSubscriptionStatus(void) 

From LSM to AIM: 

AIMInstall(IN AppID, IN AppInstallBlockFileName, OUT InstallStatus) 
AIMUninstall(IN AppID, OUT UninstallStatus) 

From CUI to LSM: 
LSMUpdateAHSubscriptionStatus(void) 
LSMGetAspList(OUT NumAsps, OUT AspID[]) 
LSMGetAspInfo(In AspID, OUT ...) 
LSMGetAppList(In AspID, OUT NumApps, OUT AppID[]) 
LSMGet AppInfo(In AppID, OUT . . . ) 

From LSM to CUI: 

AskUserYesNo(IN Message, OUT Response) 
AskUserPassword(IN Message, OUT Response, OUT Retain) 
InformUser(IN Message) 
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From CNI to LSM: 

LSMGetAppServerSet(IN AppID, OUT AppServerSet) 
LSMGetNewAppServerSet(IN AppID, OUT AppServerSet) 
LSMGetSLiMServerSet(IN AppED, OUT SLiMServerSet) 
LSMGetNewSLiMServerSet(IN AppID, OUT SLiMServerSet) 



LSM InterfacesTo/From Server Components 

From ASPWebServer to CBM: 

ASPNotify(INUserName, IN SLiMServerSet) 

From LSM to ASPWebServer: 

PromptASPNotify: HTTP POST to server cgi script to engender an ASPNotify 

From LSM to SLiMServer via CNI: 

GetSubscriptionList(IN UserName, IN Password, 

OUT RetumCodes, OUT NumberOfSubscriptions, 
OUT SubscriptionID[]) 
GetLatestApplicationInfo(IN Subscripti onID, 

OUT RetumCodes, OUT Upgradelnfo) 
AcquireAccessToken(IN SubscriptionID, IN RootFileNumber, 
IN UserName, IN Password, 

OUT RetumCodes, OUT AccessToken, OUT RenewalFreq, 
OUT AppServerSet, OUT Upgradelnfo) 
RenewAccessToken(IN AccessToken, IN UserName, IN Password, 

OUT RetumCodes, OUT AccessToken, Out RenewalFreq, 
OUT AppServerSet) 
ReleaseAccessToken(IN AccessToken, IN UserName, IN Password, 

OUT RetumCodes) 
RefreshAppServerSet(rN AccessToken, IN BadQos, IN NoService, 
OUT RetumCodes, OUT ServerSet) 

LSM Functionality Overview 

The web pages set up at \\fserv2\home\webpages\pinain.html are intended to provide a 
framework for discussing overall issues related to subscribe/install/uninstall scenarios, 
and I look forward to reviewing them with everyone, soliciting feedback, & updating the 
pages & requirements document appropriately. The material in this section is presented 
at a much lower level than the web pages. 

Run an Application 

• ECM invokes LSMNotifyAppStart whenever there is a transition from 0 to 1 
open files wrt a particular eStream app. ECM can determine the AppID of an 
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eStream app by inspecting the fully specified eFSD filename; the first level of the 
directory path is a string version of the AppID. If LSM does not recognize this 
AppID for some reason, an appropriate status is returned to ECM. 

• LSM sends an AcquireAccessToken call to via CNI to a SLiMServer [possibly 
only if the ExpectedExpirationTime for the AppID in question is in the past] and 
when LSM receives the AccessToken, it places a copy of it in the Subscription 
Data Set, assigning ExpectedExpirationTime according to the RenewalFrequency 
indicated by the SLiMServer and setting up a timer alarm to go off shortly before 
the AccessToken is expected to expire. LSM marks the AppID as ActivelnUse in 
the Subscription Data Set and returns an appropriate status to ECM. 

• Should an AccessToken be denied due to a mandatory upgrade, LSM retries 
AcquireAccessToken w/ the new RootFileNumber, updates the RootFileNumber 
in the associated Subscription Data Set entry, & calls InformUser("Upgrade"). 
The RootFileNumber for the specified AppID is always returned to the ECM as 
an OUT paramater of LSMNotifyAppStart so that ECM can take appropriate 
action if the application version has changed from that being cached. 

• Should an AccessToken be denied because it is currently held by another of the 
user's clients, a copy of the other client's AccessToken is returned. If the user 
responds affirmatively to AskUserYesNo("Yank token?"), then LSM will call 
InformUser("Please wait"), send the token to ReleaseAccessToken, & then 
retry AcquireAccessToken at the appropriate time. 

• CNI invokes LSMGetSLiMServerSet to obtain SliMServers for AccessToken 
requests. Should CNI have quality of service issues with those servers, it calls 
LSMGetNewSLiMServerSet; LSM uses the PromptASPNotify interface to 
engender the ASP Web Server to send a message containing a SliMServerSet. 

• Whenever an AccessToken timer alarm goes off, LSM checks that the associated 
application is marked ActivelnUse and if so, it issues a SLiMServer request to 
RenewAccessToken and records the returned token in the Subscription Data Set, 
resetting ExpectedExpirationTime and scheduling another timer alarm. 

• ECM invokes LSMCheckAccessToken whenever it receives an eFSD request 
(whether the associated data is currently in ECM's cache or not). If LSM does 
not have an AccessToken or if ExpectedExpirationTime is in the past, LSM 
invokes AcquireAccessToken. If ECM needs to fetch the requested data from an 
AppServer, it invokes LSMGet AccessToken. 

• CNI frequently calls LSMGetAppServerSet to obtain a set of AppServers 
through which to cycle; LSM returns the servers associated with the most recently 
acquired AccessToken. Should CNI be dissatisfied with the quality of service 
associated with the AppServers it is given, it calls LSMGetNewAppServerSet 
which sends RefreshAppServerSet to a SLiMServer. 

• Should an CNI request be denied by an AppServer because the submitted 
AccessToken was expired, CNI returns an appropriate status to its caller. If the 
caller was attempting to satisfy an access from an ActivelnUse application, it will 
undoubtably call LSMGetAccessToken. 

• ECM invokes LSMNotifyAppStop whenever there is a transition from 1 to 0 
open files wrt a particular eStream app. LSM sends a ReleaseAccessToken call 
to a SLiMServer [possibly only if the ExpectedExpirationTime for the AppID in 
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question is not in the past]. LSM marks the application as Inactive in the 
Subscription Data Set, which indicates that automatic AccessToken renewal 
should not be continued for this application [LSM may cancel outstanding alarm]. 



Get/Refresh Subscription Information 

• Whenever the CBM receives an ASPNotify from an ASP Webserver, it sends an 
LSMAspAccountPing message to the LSM. The LSM checks its ASP Data Set 
to see if the specified ASP is one that it recognizes. If LSM does not recognize 
the ASP, it allocates a new ASP Data Set entry and calls AskUserPassword to 
prompt the user for the password associated with this ASP+UserName (asking the 
user if the password can be saved on the client for future use). If LSM does 
recognize the ASP, it only calls AskUserPassword if the user did not allow the 
password to be previously recorded. 

• Once LSM has the UserName & Password, it issues a GetSubscriptionList 
request to one of the ASP's SLiMServers. For any SubscriptionlDs associated 
with that ASP that are installed on the client but do not appear in the returned list, 
the user is informed that he cannot run the app at the current time & is asked if he 
would like it uninstalled from his client (note that we may be legally required to 
remove it from the cache, so this may not be a question but rather just a message); 
AIMUninstall is invoked to achieve this. 

• For all other SubscriptionlDs, a GetLatestApplicationlnfo request is sent to a 
SLiMServer. 

• For SubscriptionlDs already in the Subscription Data Set, the RootFileNumber is 
compared to see if the application has been upgraded. If it has, the user is 
informed; the new RootFileNumber is recorded for the next run (assuming either 
that the upgrade is mandatory or that the user indicated he wanted it). 

• For SubscriptionlDs that were not already in the Subscription Data Set, a new 
Subscription Data Set entry is created and AskUserYesNo("InstaIl App?") is 
invoked. If user responds in the affirmative, LSM calls ECMMount to notify 
ECM to prepare to service requests on behalf of this new application & then 
invokes AIMInstall, which can use the eFSD/ECM path to obtain the 
AppInstallBlock. If user responds in the negative, he is able to indicate "don't ask 
again" or "ask again later", which is recorded in the Subscription Data Set. 

• Whenever the eStream client software starts up or at specific times requested by 
the user via the ClientUI, LSMUpdateAlISubscriptionStatus is called to refresh 
the eStream client's knowledge of the apps to which the user is subscribed. 

Display Subscription Information 

• The CUI provides a user interface to display ASP and application info, using 
LSMGetAspList, LSMGetAspInfo, LSMGetAppList, LSMGetAppInfo. 
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Client Browser Module Overview 



The CBM is an eStream client component that runs in association with the client browser. 
A must-have item is that it be able to communicate a small amount of data in some way 
to the core eStream client software. A strongly-desirable item is that it be realizable on 
both Internet Explorer 4+ and Netscape Navigator 4+. A would-be-nice item is that it be 
able to start the eStream client software if it were not already running. 

A number of technologies are currently under investigation. I believe that all three of the 
items above can be met by native code distributed as a browser plugin, which registers to 
handle a particular MEME encoded filetype (*.estream) in the HTML stream. Bhaven has 
created JavaScript that can start Excel & communicate with it, though my understanding 
of this solution is that it required that the browser be Internet Explorer, that the app be 
ActiveX, and that the security setting on the browser be at minimum setting. 



Issues 



• 



Decide how to store ASP Data Set and Subscription Data Set in nonvolatile 
memory on the client system. 

Develop mechanism for implementing client browser module. 
Refine needs with respect to Client User Interface. 
Add Prefetcher interface. 
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eStream App Server Low Level Design 

Version 1.2 



Sameer Panwar 




Functionality 



First, some definitions: 

eStream page : the smallest unit of data that can be requested by a client from an App 
Server. Proposed to be 4kB for eStream 1.0. 

page set: simply, a sorted list of eStream pages, each identified by a File ID (i.e. AppID 
& File #) and page # (essentially an offset into the file). This set is restricted only in that 
all pages in the set must have the same AppID. 

client request: a single self-contained message from a client requesting a page set from 
the server. Each server response to a client request can return a number of pages, and 
there is a maximum number of pages that the client can request in this message. (TBD, 
somewhere between 8 and 20 or so). 

The primary job of the App Server is to service client requests for application data blocks. 
The App Server is designed to minimize the amount of CPU time it must consume to sat- 
isfy each client request, thereby maximizing scalability. Thus, authentication is per- 
formed by a simple expiration time check of an AccessToken provided by the client, and 
compressed application data is saved persistently. 

The App Server serves data derived from eStream Sets. To decouple the performance 
needs of the App Server from the Builder, we should have a post-processing tool that 
converts the flat, uncompressed eStream Sets as provided by the Builder into a precom- 
pressed format suitable for memory mapping, if the App Server is configured to serve 
compressed bits. Also, a profiling part of the App Server can be used to monitor for 
common page sets, and then assemble more optimized replies, which compress the set of 
pages together as a unit, to take advantage of improved compression ratios. These replies 
can be stored on disk to save time in rebuilding them each time the server is started up. 

The App Server (AS henceforth) views an eStream Set as simply a set of files, and knows 
no further underlying structure. Thus an eStream Set contains at the start a table (FOST) 
indexed by File #, and providing the offset into the eStream Set where the associated file 
data begins, and the size of the file. So the AS just takes the client request of (AppID, 
File #, Page #, no. of pages), maps AppID to an eStream Set and looks up in the FOST 
table (File/Offset/Size Table) to find the requested data. 
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This works slightly differently when the eStream Set file has been pre-compressed by the 
post-processing tool The resulting image is the same as before, except now the FOST 
points to another table, the POST (Page/Offset/Size Table). Because the compressed 
pages will be of different sizes, this table must by indexed by the Page # to find the rela- 
tive offset and size of the compressed page data for the file. Thus if an AS is not config- 
ured for data compression, the main difference in behavior is that it doesn't do a POST 
lookup and it doesn't care about coalescing page sequences. 

Data type & Data structure definitions 

Processed eStream set - this structure is kept on disk and never changes after installation. 
It looks like: 

struct { 

ApplicationID appID; /* for reference, is a 128-bit GUID, see ECM 

LLD */ 

uint32 maxFileNo; 

boolean compressed_f lag ; /* indicates whether the AppFiles are com- 
pressed, though maybe we should do it differently? */ 
FOST_Entry FOST [<maxFileNo>] ; 

uint8 appData[<sum of all AppFile sizes, which are variable>] ; 
} ProcessedEstreamSet ; 

Since the files in the application are of variable size, we can't make a table out of them, 
and must indirect out of a table (indexed by the File #) to find their offset location inside 
the AppData buffer. 

struct { 

uint32 offset; 

uint32 size; 
} FOST_Entry; 

When the processed eStream set is compressed, then we use the AppFileCompressed 
structure at the offset indicated by the FOST, otherwise we interpret the data as just 
AppFile. The AppFileCompressed structure starts with a table that indicates the size and 
offset of the compressed data that belong to the page it was indexed by. 
struct { 

uint8 f ileData [<size from FOST_entry>] 
} AppFile; 

struct { 

POST_Entry POST[<number of pages, derived from size from FOST_Entry>] ; 
uint8 f ileData[<sum of all FilePage sizes, which are variable>] ; 
} AppFileCompressed; 

struct { 

Uint32 offset; 
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uint32 size; 
} POST_Entry; 



This covers all the structures that live on disk. When we mmap-per-file, that means we 
make multiple mappings out of a single ProcessedEstreamSet file, at different offsets, one 
for each file. 

Now, for the in-memory data structures (assuming per-file-mmapping): 

The primary lookup will be a hash table, hashed on the AppED and FileNo. It should 
have on the order of 10,000 entries, each table entry containing a list of entries (for colli- 
sions). Each list entry contains: 

struct { 

ApplicationID appID; 
uint32 fileNo; 

uint32 size; /* size of the mapped Appfile */ 
MMap fileMap; 
HTListEntry * next; 
} HTListEntry; 

The Mmap struct just contains any OS-specific-related stuff to manage the mappings, 
plus a field char * ptr, which points to the place in memory that the AppFile (or 
AppFileCompressed) is mapped. So the hash table looks like: 
struct { 

HTListEntry * entry[<size of hash table>] ; 
} MMapHT; 

Hash function is TBD. The hash table should be statically sized large enough to handle 
the full number of eStream sets up to the maximum memory we will support. Assuming 
32 bytes being used per entry, that implies about 1 MB to handle 30k files, which is no 
problem. (Maybe we should reserve entries for 100k files or more?) 

Configuration: Each AS must obtain configuration data, either directly from the data- 
base or from the monitor in its startup message. The required data is (with the config pa- 
ram names and datatypes): 



AppList 
ServerPort 
Monitor Port 
SLiMKey 
ClientTimeOut 



vector of ApplicationID' s (128-bit GUIDs) 

uintl6 

uintl6 

uint (size TBD, depends on actual algorithm) 
uint32 



CompressionFlag uint32 



Network communication : The AS talks only to clients and the server monitor via the 
network. The server monitor communication will be described as part of the monitor 
heartbeat protocol. The AS-client communication will be described in a separate docu- 



Omnishift Technologies, Inc. 



3 



Company Confidential 



eStream <COMPONENT> Low Level Design 



ment. The AS will time-out and close connections that have been idle for some amount 
of time (a few seconds). 

[maybe combine multiple responses into a single send socket call (will only work for 
TCP probably, since proxies won't like multiple server responses)?] 



Interface definitions 



The AS is optimized to do one thing only: serve pages from the read-only file system pa 
of eStream, so there is just one interface with the client. Anything the client can care 
about in an eStream set is just another file to the AS, including the AppInstallBlock, and 
directories/metadata. The AS only returns the data the client requested, nothing extra. 

struct { 

uint32 fileNo ; 

uint32 pageNo; 
} PageRequest; 

struct { 

uint32 errorCode; 
uint32 compressedFlag; 
uint32 fileNo; 
uint32 pageNo; 

uint32 offset; /* offset into pageData below */ 
uint32 dataSize; 
} PageReply; 

PageReadRequest 

Caller: Client 
Callee: AppServer 



Input: 



uint32 appld; 
eStreamAccessToken accessToken; 
uint32 numPagesRequested; 
PageRequest pageSet[(numPagesRequested)] ; 



Output: 



uint32 
PageReply 



numPagesRequested; 
pageSetReply[(numPagesRequested)]; 
pageData[(sum of all page data)]; 
globalErrorCode; 



uint8 
uint32 



Global Errors: INVALID_ACCESS_TOKEN 
EXPIRED ACCESS TOKEN 
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INVALIDAPPJD 

EXCEEDEDMAXJREQUESTABLEPAGES 

Errors within 

PageReply: INVALIDFELENO 
INVALIDPAGENO 

SERVER ERROR (probably should be logged, and should cause an alert if too many 
occur in some time period, including errors that don't get returned to the client.) 

AppServers don't ever talk to the database (it would be a waste of licenses considering 
the number of AppServers we'd have and their infrequent accesses). Instead, they obtain 
all their relevant control information from the server monitor. 

The exact interfaces are TBD, but from the monitor they will provide configuration in- 
formation, AppServer state change requests, and add/remove requests to the list of apps 
being served. Going back from the AppServer to the monitor, it will report load (average 
response time) on a per app basis, and server state, along with the heartbeat. 



Component design 

Interesting issues to deal with: 
Scalability/Performance 

Since scalability (and thus performance) is critical for the AS, let's go over how CPU and 
memory are used. 

Memory 

Performance is maximized when virtually all client requests can be satisfied by retrieving 
the desired pages from RAM, because RAM is far faster than disk. Thus the amount of 
RAM available will put an upper bound on the number of apps that a single AS can serve 
efficiently. Since server RAM won't grow as fast as the total size of all apps available as 
eStream sets, this means we'll have to heterogenize servers, where each server specializes 
in a subset of apps, limited by available RAM. For eStream 1 .6, this component of AS 
configuration will be handled manually, the eStream administrator assigning apps to 
servers. In the future, the set of App Servers should automatically reassign apps dynami- 
cally to balance load. 

But this is just one level of memory, committing RAM to a set of apps. There still re- 
mains the question of how to best utilize that RAM for each app, since some files are 
used far more often than others. This immediately means that for efficiency we must 
overcommit RAM, because if we allocate an entire eStream set into RAM, we're using 
precious resource to hold data that may be requested only very rarely. Instead of having 
to manage our physical RAM manually to accomplish this (such as with a cache), an eas- 
ier approach would be to take advantage of virtual memory (VM) to automatically keep 
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the hot pages in RAM, with the remainder available (again automatically) off disk (via 
memory mapping the eStream sets). That way the server can satisfy any possible client 
request for any app it serves, but is optimized to be the most efficient over all clients. But 
this only works if enough VM is available. (Time for some back-of-the-envelope num- 
bers.) Given that an app seems to have something like only 20% of it being hot (from our 
current limited data from the prototype), this means VM must be at least 5x of RAM for 
maximum efficiency. Given that a process has about 2 GB addressable VM, this corre- 
sponds to about 400 MB of RAM. Beyond that size (which is not uncommon), we don't 
have enough VM to efficiently overcommit our memory (by mmaping entire eStream 
sets). So now our choice is to either manually manage a memory cache (and all the atten- 
dant coding, bugs, etc.), or to mmap at a finer granularity. 

Note that the effective virtual memory required by an app is increased when compression 
is used, to handle the extra compressed page sets. They'll probably double or triple the 
RAM footprint by hot pages (due to redundancy), but only increase the overall VM foot- 
print by 1.2 - 1.5. The consequence of this is that the overcommit ratio goes down to 1 .5 
/ (3 * .2) = 2.5, though the amount of apps servable is reduced to 1/3 (!!). Now 2 GB vir- 
tual address space corresponds to 800 MB of RAM. This means we should be able to just 
memory map entire eStream sets, up to 2 GB worth, and be confident we're utilizing 
RAM efficiently, assuming the server has about 800 MB of RAM. A server with less 
RAM will likely thrash, and those with more will likely see little improvement in the 
number of apps they can serve via memory mapping. 

A loss of 2/3 in the number of apps an AS can serve I think is too great a sacrifice, too 
great a loss in app scalability (need 3x the number of servers as before!) for what is about 
a 15-30% greater effective bandwidth at the client. The root of this problem is the redun- 
dancy (costly in physical memory), because the compressed page sets will contain the 
same page in multiple sets. This is similar to the redundancy that appears in trace proces- 
sors and dynamic translation, which places extra memory demands in both those cases. I 
think we must completely eliminate this redundancy to achieve the goals we desire, either 
by (1) not using compressed page sets, and just sending multiple individually compressed 
pages, or (2) ensuring a page appears in only one compressed page set. [There further 
potential loss of effective memory size when using compressed page sets since they'll be 
allocated in 4k chunks, thus wasting about 2k on average; we'd have to batch them up 
together in files to minimize this. . . Also, saving the compressed page sets to disk intro- 
duces extra complexity to the AS because we'd have to properly handle recovery (i.e. 
what if the system crashes while we're writing the sets, which if we're memory mapping 
is totally out of our control). Because of this robustness requirement, and the fact we 
need to be 100% sure we're serving good bits (lest we crash a bunch of clients), this 
needs to be thought out very carefully if we want to do this. My opinion is that we should 
defer implementing compressed page sets until we better understand the tradeoffs, and 
good profiling schemes. In particular, will the AS be mostly bandwidth-limited, memory- 
limited or CPU-limited?] 

Separately from this, we should consider the effect of per-file memory mapping (ignore 
the compressed page sets now). This has the impact of requiring many more mmap's 
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from the OS, but promises better use of the limited virtual address space. In this scheme, 
we mmap each file into VM as it is referenced by a client. If only hot files are referenced, 
then the RAM footprint is the same as before, but VM is only used for the hot files, not 
the entire app, probably about 30-50% greater in size. Thus the overcommit ratio then 
becomes 1.5, much better than the 5 with full app mmapping. So 2 GB of VM corre- 
sponds to 1 .3 GB of RAM, much better than the 400 MB with full app mmapping. How- 
ever, this assumes that VM is used in a cache-like manner, evicting not recently used 
mmap's, since as uncommon files are referenced, they eat up more and more VM. Once 
VM is totally used up, then replacement policies and eviction (and fragmentation of vir- 
tual address space) become issues, just as with a manually managed cache. One solution 
is to simply purge all mmap's and start from scratch, which is simple and reliable, espe- 
cially considering the AS is multithreaded (if this is done, the above analysis doesn't 
hold, and performance becomes a function of how often VM is cleared). Another possi- 
bility might be to use the profiling mechanism and only place sufficiently popular files in 
mmaps and do regular file system accesses for the rest. 

Of course, the alternate option for managing physical memory is to know its size, and 
manage a cache manually. One advantage here is that the AS would know the physical 
memory consumption and usage (unlike when the OS was handling everything), which 
may help with load balancing. The main advantage is that there are no artificial limits 
(overcommit ratio is irrelevant), and only physical memory size is the true limit, and this 
approach can map any number of eStream sets (with any size of files) to any amount of 
physical memory up to the virtual address size (4 GB). Then memory management be- 
comes an issue (what do you do once all your RAM is full), which can be painful in a 
multithreaded environment. Again, we can just invalidate the whole cache as an option, 
but this will probably happen more often than with the per-file-mmapping case, unless 
RAM is greater than the maximum that the per-file-mmaping approach can handle. If the 
wholesale cleanup approach is used, then allocating fixed size chunks may not be needed, 
and we could potentially get better memory usage by packing compressed pages more 
tightly (e.g. 16-byte aligned vs. 4kB aligned), which is another potential advantage. 
Maybe instead of wholesale cleanup, we mark the most commonly used pages, and then 
just compact those and dump the rest (say 50%). The main issue with this approach is 
potential redundancy with respect to the OS disk cache (which is shared in the mmap ap- 
proach), and assumption that our caching policies will be better than the OS's. Also, 
lookups get messier, since we need a bigger lookup table to index via page # as well. 

Yet another option is to use multiple processes instead of multiple threads, one process 
per app being served, thereby releasing us from the 2 GB VM limitation. However, this 
introduces the issue of multiplexing requests from the network via IPC, and more load on 
the server monitor. On x86 NT, a Very Large Memory feature is available that can pro- 
vide 36-bit addressing per process; we may want to use this even though it won't be 
available on regular Unixes (and probably not x86-linux). 

In summary: per-eStream-set-mmapping is probably too wasteful of virtual address 
space. Per-file-mmapping is much better, but then memory management becomes an is- 
sue, suggesting a simple throw-away-and-start-over solution. However, given that solu- 
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tion, if a lot of physical memory is desired, a manual cache approach maybe better (the 
better packing should overcome any loss due to redundancy with the OS disk cache). 
Compression of page sets invokes several issues that probably can't be fully addressed 
until after 1.0. Bottom line : the target now for 1.0 is to use per-file-mmapping with 
per-page compression (but no compression of page sets). Also, we should instrument 
the system to allow us to easily collect the relevant data (mem usage, CPU load of differ- 
ent routines, etc.) to help guide us in further evolution of the system to improve perform- 
ance (e.g. compressed page sets or explicit page cache). 

CPU 

The main work of the CPU is as follows (encryption is assumed to be done by hardware 
since its CPU impact is severe): 

1 . OS system call to retrieve request from network. 

2. Decode client request. 

3. Validate AccessToken. 

4. Lookup AppDD, File # in primary lookup hash table. (If mmapping eStream sets, in- 
stead lookup in App table, then lookup in FOST). 

5. If mmapping then (if uncompressed, no further lookup, if compressed, then lookup in 
POST to find page and size), if explicit cache then look in B-tree (secondary lookup). 

6. If lookup fails, then bring in the data off disk (either mmap or file system call). 

7. Copy page data to reply buffer. 

8. OS system call to send reply to network. 

However, if compressed page sets are used, lookups get more complicated, with a differ- 
ent set of tables to check for an appropriate page set first (and lookup failures incur poten- 
tial decompression/compression). It appears the least amount of CPU time is probably 
incurred when doing per-file-mmapping. All pages held in memory are kept in com- 
pressed form to save repeated compression of the same data, so pretty much all the work 
is in lookups and memory copies. Potentially the AccessToken validation will use hard- 
ware assist. Lookup failures (i.e. having to go to disk) should be relatively uncommon, 
and memory should be sized to ensure that. 

However, since the AS will run in user mode, this incurs the penalty of two extra copies 
(from the network buffers) and switching between kernel and user mode twice. If this is 
enough of a problem, we'll have to consider implementing the AS to run in the kernel (all 
commercial NFS, etc. implementations run in the kernel), which means we should choose 
our implementation to be compatible with that approach. In particular, we may not be 
able to rely on the virtual address space not being fragmented, so mmapping full eStream 
sets may be impossible. Plus robustness of the server becomes even more important, and 
portability issues arise. For the 1.0 release, we plan to implement the AS in user mode 
keeping the possibility of moving to kernel mode in the future, and will collect data from 
1 .0 (or derived prototype) to evaluate the actual benefits. 

Disk : Since we are relying heavily on the common pages being in memory, we could 
possibly even consider storing the processed sets on a network disk, i.e. remote from the 
app server itself. However, such sharing won't work well for compressed page sets since 
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they are written to at runtime — it would be extremely messy to handle dozens of app 
servers trying to add many compressed page sets (possibly the same) to a set of shared 
files. 



Multithreading model 

The approach will be to have a single boss thread which pulls things out of the network 
port and stuffs client requests into a queue and a bunch of worker threads which grab re- 
quests and send back the replies. Simple enough, but this raises the issue of thread con- 
trol, since the boss also needs to be able to handle threads that die or hang and kill and 
restart them. The boss thread will monitor the worker threads and provide load/hearbeat 
info to the monitor through the server manager thread, thus giving visibility to the server 
monitor of the health of all the worker threads. 

Load balancing 

To be described elsewhere? (appears in SLiM server LLD) 



Security 

There are two levels of security involved in the AS. First, we must prevent clients who 
don't hold valid licenses from gaining access to the licensed binaries. This is accom- 
plished by the client obtaining an AccessToken from the SLiM server and presenting it to 
the AS upon every request. The AS can then use the SLiM server's public key to test the 
authenticity of the AccessToken (to protect against forgeries), and then can test the au- 
thentic expiration time of the AccessToken. Second, we must encrypt the actual data be- 
ing sent on the wire to prevent third parties from gathering the binary data covered by the 
license. Since the data coming out is somewhat obfuscated anyway (files are identified 
by arbitrary IDs, with our own strange message formats and compression and all in ran- 
dom pieces, etc.) it is not clear how much extra protection is really necessary, i.e. what do 
the license issuers actually want? We should use a common scheme like SSL to perform 
this encryption. It has been decided that the encryption load for this would be too great, 
and thus the data send back will be unencrypted. We may use SSL for authentication 
purposes only (i.e. null-cipher), if that is cheap enough. 

Also, a possible optimization for checking AccessTokens would be to cache recently used 
AccessTokens along with a signature/hash. If a token presented by a client matches, then 
we can skip the authentication step (since we've done it once already) and just check the 
expiration time. 
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Robustness 

The AS must be very robust. It must catch OS call errors and handle/log them as appro- 
priate, and deal with threads that hang or die. Thus it needs to aggressively check for er- 
ror conditions and possible failure modes. The AS also needs to track relevant resources 
(e.g. sockets, memory) and carefully manage/reclaim them so as not to exceed any limits 
or to degrade performance. And of course, the AS needs to check all data coming in from 
the client, to deal with ill-formed requests, and illegal values (e.g. huge negative indexes, 
etc.), and perform no potentially dangerous operation without validating parameters. This 
becomes even more important when we eventually move the AS to run in kernel mode. 
The AS also needs to be as stateless as possible, to minimize recovery time, and if it does 
perform writes to disk (such as for the compressed page sets), do so in a reliable fashion 
conducive to quick recovery. Any unreliability in the AppServer will negate any benefit 
of scalability we have over our competitors. 



Testing design 

This document must have a discussion of how the component is to be tested. Some sub- 
sections could include: 

Unit testing plans 

The various components of the AS are not too large or complicated: The request dis- 
patcher (to worker threads), the hash table, the compression code, the AccessToken 
checking code, etc. These shouldn't be too hard to do reasonable testing on in isolation. 

For the post-processor component, we'll have to build some sample Estream Sets as in- 
put, but it'll be hard to tell whether the output is correct without having a minimal work- 
ing AS. 

Cross-component testing plans 

The best approach will be to perform incremental implementation and testing. I.e. we 
build the core functionality that is required (i.e. can start with just regular i/o reads), and 
then add the more performance-related stuff later (adding mmaping, and then the hash 
table & AccessToken checks), while testing the entire system as pieces are gradually 
added (of course performing sanity-check and other minimal testing on the pieces first if 
possible). Compression can be added last. 

To actually drive the AS, we'll need a test client, which will be designed to just shoot off 
a series of read requests to the server. The file data returned could (hen be written to 
files, and this can be compared against the original set of files used to create the Estream 
Set we started with, to check that the data was received properly. For checking error con- 
ditions, a log of errors can be written and compared against a reference log for those re- 
quests we expect to fail. 
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Stress testing plans 

To accomplish this, we should run multiple independent test clients (on the same machine 
and on different machines), and increase the frequency of requests (to stress the AS's 
threads and sychronization, and communication routines), and the number and size of 
files referenced (to stress the hash table and memory). Each test client can then check 
whether the data and errors it got back were as expected, like in the above subsection. 

Coverage testing plans 

Should we use some kind of code coverage tool for this? 
Performance testing plans 

Since performance is critical, we should take the time to evaluate the AS's performance 
characteristics. We need to crank up our stress testing until either bandwidth or CPU 
saturates, and record the request rate that generated it. We should compare how this point 
responds to high numbers of clients with fewer requests per client vs. fewer clients with 
higher requests per client. We'll need to profile the system to find bottlenecks to tweak 
more performance out of it, and learn how well our original design assumptions hold up. 
Depending on whether CPU or bandwidth (or memory) saturates first, we may want to 
modify the system's tradeoffs to improve scalability further, and otherwise note which 
components a customer should upgrade for better performance. Also, if we think we can 
come up with reasonable client access pattern profiles, we may want to use those to esti- 
mate the actual number of real-world clients an AS can support. As part of this, we'll 
probably want to run the AS in-house once it is mature enough (eat our own dogfood), 
and then farm out app upgrades, etc. (play out some of our scenarios) and see what hap- 
pens to the AS's (do they choke or what). 

Availability testing plans 

We will also need to test our failover and load balancing capabilities. This will require 
several test machines with the monitor in place to start and stop servers, and have clients 
be aware of multiple AS's and respond appropriately when an AS stops responding. For 
load balancing, we'll probably want a bunch of test clients with a variety of access pat- 
terns and see how well their requests are distributed. 



Upgrading/Supportability/Deployment design 

App Servers will possibly need to version their interface with clients (requiring clients to 
state the version they're expecting), but will also need to support older versions. We may 
also modify the Estream Set format (or just the processed set format), but that should be 
handled by upgrading both the AS and post processor and then regenerating the processed 
sets. 
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For supportability & deployment, the AS will report error conditions and load to the 
server monitor, which is used by the customer. 



Open Issues 

1 . Is there a limit to the # of possible mmaps? 

2. Is there a single system call to unmap all mmaps? 
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eStream Server Component Framework 
Low Level Design 



Michael Beckmann 




Functionality 

The Server Component Framework provides a common basis on which server compo- 
nents are implemented. The framework provides a number of services such as common 
server initialization and configuration, messaging, state management, logging, and error 
handling. The component framework ties together many of the core utilities provided for 
the server components. 

The advantage of the framework is that heterogeneous server components can be man- 
aged in a consistent manner with the expectation that all server components will commu- 
nicate and behave consistently within the system. 

All server components with the exception of the web server will be built on top the 
Server Component Framework. To make use of the Server Component Framework, a 
specialized server component will need to extend the framework by implementing the 
methods high-lighted in gray. Implementing these interfaces makes the specialized server 
component "plug-able" within the framework. 
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The following table give a brief description of each of the routines that need to be spe- 
cialized by each server component to make it plug- able into the Server Framework: 



StartProcessing 


Specialized server component routine to request the server compo- 
nent to start processing work. 


StopProcessing 


Specialized routine to request the server component stop processing 
work and transition into an idle state 


UpdateConfig 


Specialized routine to dynamically update configurations while a 
component is either in the processing or idle state. 


HandleError 


Specialized routine to handle the occurrence of an error 



Server State Manager: 

At the heart of the server component framework is the Server State Manager. The server 
state manager is a set of interfaces that initiate and manage state changes within a server 
component. All Server components, by virtue of being built on top of the component 
framework, can be managed uniformly across a deployment. 



The Server State Manager implements a simple state machine that is shared between 
components. It manages the state transitions within the server component. Additionally, 
the state manager maintains current state information for each server component and logs 
state transition history in the event that a server component terminates unexpectedly. 

As specified above, each server component is required to implement a number of 
transition methods, with pre-defined signatures, which the state manager will execute 
when making a state transition. 

The following diagram shows the state diagram and the associated transitions: 



Omnishift Technologies, Inc. 



2 



Company Confidential 



eStream Server Component Framework Low Level Design 



Stopped^^- 



/ StopServer 



/StartServer 



/ RaiseError 



/ StopProcessing \ / 

^Processing^ Idle ^ 



7K 



I UpdateConfig 
I 



~~ J^ I StartProcessing j " 



/ RaiseError 
I 




/ RaiseError 



n — J 



Message Service: 

The Server Component Framework depends on a message service which is used by the 
Server State Manager and Configuration Manager to communicate with the System 
Monitor. 



The Server State Manager uses the messaging service to listen for state change requests 
from the System Monitor which it satisfies by returning the current state, any up-to-date 
status, and load information. 

The Configuration Manager uses the message service to request configuration informa- 
tion from the System Monitor. Although each server component could easily go to the 
database for configuration information, it has been decided to go through the monitor as 
to save db licensing costs. 

See below for more details on messaging protocols for the Server State Manager and the 
Configuration Manager. Also, refer to the low-level design document for details on the 
design of the eStream Messaging Service (EMS). 

Configuration Management: 

The configuration management utility is used by all server components to manage the 
server configurations. It provides the following functionality: 

• Configuration for a server consists of a set of name - value tuples where the val- 
ues themselves can be a set of name-value tuple. 

• Servers can load the complete configuration from the database (indirectly). 
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• Servers can load the configuration for a given name. 

• Servers can load the configuration from a flat file also. 

On the Server Manger interface, configuration will appear as a table containing name - 
value tuples. The table may be hierarchical to represent nested structures containing the 
values which can themselves be name values. An example of a simple name- value pair 
would be: 

port 8080 

An example of nested name values would be: 

Applications: 

word, exe windows2000sp3 
excelexe win98sp4 

On a flat file the configurations will always be name-value pairs. To represent one level 
nested structure the format would be: 

Applications word.exe windows2000sp3 
Applications excelexe win98sp4 

A common set of configurable parameters is defined for all server components. These 
configurations are maintained by the Server Component Framework in collaboration with 
the Configuration Manager. All configuration information is persistently stored within 
the database. The common configurations are used to initialize the server component 
after the component process has been launched. Refer to the configuration table below 
for more details on common configurations. Specialized server components can support 
additional configurations (non-common) depending on the server type. These 
configurations are read from the database and updated when a server component starts 
processing. They can also be updated dynamically while a server component is 
processing through the use of the UpdateConfig interface. 



The list of common configurations include: 



Information 


Supports 
Dynamic 
Config 


State 


Description 


ServerlD 


No 




Unique identifier for server components. This 
server identify is unique within a deployment. This 
ServerlD is not known to eStream clients. Its pur- 
pose is as a handle to uniquely identify server com- 
ponents. 


ServerType 


No 




Identifies the type of server component. One of the 
following applies: 
■ Primary Monitor 
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■ Backup Monitor 

■ Application Server 

■ SLiM Server 


DbUser 


No 




User name string required for database connectivity 
for this server ID 


DbPasswd 


No 




Database password associated with the DbUser 


Dsn 


No 




Data Source Name used to access the database. 


PortNum 


No 




PortNumber used for light-weight messaging lis- 
tener 


MachinelD 


No 




Machine TD is used to eet at important machine in- 
formation needed for all server components such as: 

■ IP address for the machine server component is 
hosted on 

■ Domain name for the machine 

■ Machines name 


AutoReStart 


Yes 


Any 
State 


Flag indicating that server component process can be 
restarted automatically without manual intervention. 
This info is consumed by the System Monitor. 


HeartBeat- 
TimeOut 


Yes 


Any 
State 


Specifies the timeout period for the listener. If the 
timeout neriod reached The comnonpnt a^ < ?iimp<s 
that it has lost the connection. All Server compo- 
nents have a listener by which they receive instruc- 
tions from the primary system monitor. Even the 
monitor has a listener that communicates with the 
Server Admin UI. 


HeartBeatRate 


Yes 


Any 
State 


Frequency at which the heart-beat is sent to this 
server component. Specified in milliseconds. This 
item is consumed by the System Monitor. 



Command Line Utilities: 



The Command Line Utilities component provides a consistent way to define and process 
command line arguments. To use this utility, the using component must define a table of 
arguments, which defines the valid set of arguments, whether or not they are required, 
and any default values. 

Arguments are specified on the command line as name/value pairs. The utility imple- 
ments the following command line syntax to support the name/value pairs. The argument 
syntax is defined as follows: 



<name>=<value> 



name 


Name is an alpha-numeric identifier. The Name can be of arbitrary length as 
supported by the system however shorter names are recommended. Names are 
case sensitive 


value 


Any alpha-numeric value. Punctuation characters may also be used. Values are 
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case sensitive 



There can be no spaces between the <name> "=", and the <value> elements. The exis- 
tence of one or more spaces or tabs delineates separation between arguments on the 
command line. 

Example: server.exe sid=267 dsn^oracle user=michaelb passwd=mypasswd 

□ If a named argument is specified more than once on the command line, subse- 
quent arguments will cause a diagnostic to be issued and the argument will be ig- 
nored. 

□ This utility allows the user to specify default values for arguments. If a default 
value is defined then the argument will be processed with its default in the event 
that the argument is not specified on the command line. 

□ This utility allows the user to tag specific arguments as required. If the required 
argument is not specified on the command line this utility will raise a diagnostic 
for the required argument. Not specifying a required argument will cause a fatal 
error. 



The following options are supported: 



sid 


Server Component Identifier. Each server component within a deploy- 
ment is uniquely identified via the sid. The sid is a handle into the data- 
base for accessing information unique to a specific server component. 


dsn 


Data Source Name. A data source name is necessary to establish an 
ODBC connection. Data Source Names are generated by an ODBC ad- 
ministrative tool 


dbuser 


User name. For database access security, all components need to connect 
as a specific user. 


dbpasswd 


password associate with the dbuser 



Logging Utilities: 



All servers and clients in eStream 1 .0 need to log the error and access data. Logging en- 
ables component debugging and auditing support. 

EStream Framework should provide logging with the following features: 

• Each component will have an error and optioanally an access log file. The names 
of these files would be <component>_error.log and <component>_access.log. 

• The files will be located in the <eStreaml .0 Root Dir>\logs directory. 

• The error log files will have messages with the following priorities: 

o 4-Low : A warning which can be ignored. 

o 3-Medium: A warning which needs to be looked into. 

o 2-High: Recoverable Error in the component. 

o 1 -Critical: Fatal Error. Needs admin assistance. 
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• Logging level should be configurable. The following levels are to be supported. 

o 0: Only errors will be logged. This will be the default level, 
o 1 : Errors and Warnings to be logged. 

o 2: Errors, Warnings and Debugging information to be logged, 
o 3: Errors, Warnings and advanced Debugging (like memory dumps, tcp 
stack dumps etc) to be logged. 

• Log Wrapping to be supported. The log files will wrap at a predefined size. On 
wrapping the following actions will occur: 

o Any existing <logfile>.bak will be deleted from the system, 
o The current <logfile> will be backed to <logfile>.bak 
o The component will continue logging to the <logfile>. 



For each eStream client and server component logging the log files (component_error.log 
and component_access.log) should be written in eStream 1 .ORootMogs directory. The 
formats for the log files will be as follows: 



Error Log: 



[HEADER] 

[TimeStamp] [Thread ID] [Priority] [Message] 
[FOOTER] 

An example of this log format would be: 



Omnishift eStream Application Server 
Server Started. 

StartTime: ^0HMMMpfc :31 :19 -0700 
IP Address: 1.1.1.1 
Logging Level : 3 

********************************************** 



[^^llS§ill*B^ 16:31:19 -0700] 0 2-High Cannot connect to the database. 
Invalid Use rname/ Pas sword . 

Mfc^|llfc ::L6:31:19 -0 70 °1 1 l-Critical Cannot start the HTTP listener 
at port 80. 

[l4/Aug/2000:16:31:19 -0700] 0 l-Critical Shutting down the server. 



Omnishift eStream Application Server 

Server Stopped..^ 

StOpTime:^^^^^^: 16:35:19 -0700 
IP Address: 1.1.1.1 
Logging Level : 3 

********************************************** 
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Format of Access Log Message: 
[HEADER] 

[TimeStamp] [Thread ID] [Message] 
[FOOTER] 

Data type definitions 

Server State: 



The server components can be in any one of the following states: 



State 


Description 


STOPPED 


If a server is in the STOPPED state then the component 
process is not running. 


IDLE 


Server component is up and running. The server has been 
initialized with the common configuration and the messag- 
ing system has been enabled. 

The listener is actively waiting on the System Monitor for 
transition requests. 

The server component is not processing any work specific to 
this servers specialization. 


PROCESSING 


Server component is actively taking requests and processing 
work specific to its specialization, ie. serving access tokens, 
and application file requests. 


ERROR 


An error has occurred in the system. Unless the server 
component is configured with AutoReStart and ERROR 
state must be manually cleared by the server-side adminis- 
trator. 



Server State Transitions: 



Changes in server component state are initiated either by the System Monitor or directly 
by the server-side administrator for the system monitor. The exception to this is when an 
error condition is raised by a server component. In this case, the component will initiate 
the state change itself. The following state transitions are supported: 



Action 


Description 


STARTSERVER 


Server is expected to be in the STOPPED state. 
If a server component is configured to support AutoReStart 
then the ERROR state is also a valid state from which to initi- 
ate this action. 


STOP_SERVER 


Causes the server to exit its process. The server can be stopped 
from any state. 
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START_PROCESSING 


Causes the server to change from the IDLE state to the 
PROCESSING state. 


STOP PROCESSING 


Causes the server to change from processing to IDLE state. 


UPDATECONF1G 


Request that the server read its configuration from the configu- 
ration manager and change its configuration. 


RAISEERROR 


Request that the server go to ERROR state. This causes an er- 
ror handler to be called. If the error is fatal it will cause im- 
mediate termination of the server process. 


Finite State Table: 



FSMTableEntry ServerStateMgr::FSMTable[] = 
{ 

{ START, { {START_SERVER,STOPPED, START_SERVER,NULL} , 

{STARTPROCESSING, STOPPED ,START_PROCESSING, NULL}, 
{NULLREQUEST, NULL_STATE, NULLREQUEST, NULL} } }, 



{STOPPED, { {STARTSERVER, IDLE, NULL REQUEST, &StartServer}, 

{START_PROCESSING, IDLE, STARTPROCESSING, &StartServer}, 
{RAISEERROR, ERROR, NULL REQUEST, &HandleError} , 
{NULL REQUEST, NULLSTATE, NULL REQUEST, NULL} } }, 

{IDLE, {{START_PROCESSING, PROCESSING, NULL REQUEST, 
&StartProcessing} , 

{STOP_SERVER, STOPPED, NULL REQUEST, &StopServer}, 
{RAISE ERROR, ERROR, NULL REQUEST, &HandleError}, 
{UPDATE CONFIG, IDLE, NULL_REQUEST, &UpdateConfig} , 
{NULL REQUEST, NULL STATE, NULL REQUEST, NULL} } }, 

{PROCESSING, {{STOP_PROCESSING, IDLE, NULL_REQUEST, 

&StopProcessing} , 

{UPDATE_CONFIG, PROCESSING, NULLREQUEST, 

&UpdateConfig}, 

{STOP_SERVER, IDLE, STOP_SERVER, &StopProcessing}, 
{RAISE ERROR, ERROR, NULL REQUEST, &HandleError}, 
{NULL REQUEST, NULL STATE, NULL REQUEST, NULL} } }, 

{ ERROR, {{STOP_SERVER, STOPPED,NULL_REQUEST, NULL}, 

{NULL REQUEST, NULL_STATE, NULL REQUEST, NULL} } } 

{NULLSTATE, { {NULLREQUEST, NULLSTATE, NULLREQUES T, 

NULL} } } 

ii : 
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A light-weight messaging protocol is needed to facilitate communication between server 
components. The primary purpose of the messaging protocol is to communicate transi- 
tion requests to the server components. In response, server components communicate 
state, status, and load information back to the System Monitor, 

The messaging protocol supports two primary message types. 1) Requests for the System 
Monitor to perform on other servers. 2) Requests to the server components themselves. 
These message types are distinguished through the protocol as described below. If the 
receiver ID and the target ID are identical then the request is for the receiver. If the target 
is different than the receiver, the message is for the System Monitor to enact a request on 
the target component. 

All requests are required to be acknowledged. Without an acknowledgement the message 
is considered un-received. 



OpCode 1 senderlD | receiverlD | targetID | Data 



The following table describes the protocol used by the Server State Manager in its com- 
munication with the System Monitor. 



OpCode 


Description 


Data 


0x01 


Request for current state 


None 


0x02 


Acknowledgment 


■ Current state 

■ Load info 

■ Status info 


0x03 


Stop Server request. Acknowledged with 0x02 
message 


None 


0x04 


Start Server request. Only valid for System 
Monitor. Acknowledged with 0x02 


None 


0x05 


Start Processing Request. Acknowledged with 
0x02 


None 


0x06 


Stop Processing Request. Acknowledged with 
0x02 


None 


0x07 


Update Configuration Request. This is a re- 
quest for a server component to request its spe- 
cialized configuration information from the 
System Monitor and update itself. Acknowl- 
edged with 0x02. 


None 



Interface definitions 



Server State Manager: 



class ServerStateMgr 

J : 
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private: 



ServerState CurrentState; 

static FSMTableEntry FSMTableQ; 



public: 



iL 



ServerStateMgr(void); 
~S erverS t at eMgr(void) ; 

ServerState SetState(ServerState); 

ServerState GetState(void); 

ServerState ProcessRequest(ServerRequest); 



SetState 



Description: Sets the current state of the server component. 

1 . Log the state change request 

2. Update the state field within the server component in memory data struc- 
tures. 

3. Send message to requester informing them of the successful state change. 
Note: SetState does not update the database directly as in the orginal design. 
The database is updated by the System Monitor once it has received an ac- 
knowledgement. A state transition is not complete until SetState returns suc- 
cessfully and the Monitor has update the database. 

Input: state value to set current state to. 

Output: current state after the new value has been set. If an error occurs will 

go to error state. 

Errors: 

1 . Invalid state argument 

2. Failure to either connect or commit state change to the database. 



GetState 



Description: returns the current state. This function does not read from the 
database to get the current state. The assumption is that if the server compo- 
nent is up and running and that it maintains a valid state. 
Input: none. 

Output: returns the current state. 

Errors: None. Will always return a valid state. 



ProcessRequest 



Description: request to the Server State Manager to change server 
state. This routine implements the guts of the state machine. 

1 . Get the current state, and transition request 

2. Index into the FSM table and continue to transition from state to 
state until the transition request is satisfied. 

3. Each state transition calls the specialized transition routines for 
each component. 

4. Call to SetState to complete each state transition. 

5. In the case of an error the state machine will process a 
RAISE ERROR request which will call the specialized Han- 
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dleError and transition to the ERROR state. 




Input: server transition request. Refer to table of valid requests de- 




fined above. 




Output: current state after the request has been completed. 




Errors: 


Server Component Framework: 



private: 



public: 



Errorlnfo* 

ServerConfig* 

Connection* 



// abstract base class 

Error; // maintains error if error was detected 
Config; // holds common configuration 
Listener; // messaging utility 



virtual int StartServer(void); // may be specialized by a server component 

virtual int StopServer(void); // may be specialized 

virtual int StartProcessing(void) = 0; // must be specialized 

virtual int StopProcessing(void) = 0; // must be specialized 

virtual int UpdateConfig(void) = 0; // must be specialized 

virtual int HandleError(void) = 0; // must be specialized 

void Run(Request); 



StartServer 



Description: Called by the Server State Manager when a server compo- 
nent is to be started. The StartServer routine is provided as part of the 
SeverComponent class. It performs the following: 

1 . Send request to System Monitor to request an update of common con- 
figuration information. 

2. Apply the configuration information to the server component. 

3. Construct a listener connection object and start the message service. 

4. Return success or failure. 
Note: 

■ This routine must return immediately to the main thread. Otherwise 
the Server State Manager will be blocked. 

■ Successful return from the StartServer routine will put the server 
into the IDLE state. 

Input: None. 

Output: Value of 0 if successful else error condition 

Errors: May return negative error condition 



StopServer 



Description: Called by the Server State Manager. 

1 . Perform any necessary cleanup. 

2. Send last acknowledgment confirming shutdown to requester 

3. Shut down the messaging system and the listener. 

4. exit process 

Note: The monitor will update the database and perform logging. 
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Input: None. 

Output: Value of 0 if successful else error condition 
Errors: May return negative error value 



StartProcessing 



Description: Called by the Server State Manager. This routine must 
be defined by each specialized server component. This routine is 
used to provide all functionality unique to different types of servers. 
1 . Spawn a primary processing thread (also known as the boss 
thread). 

a. Read server specific configurations unique to this type of 
server component from the System Monitor 

b. Spawn worker threads. Depending on the server type this 
routine does the heavy lifting to either process access to- 
kens and renewals in the case of SLiM server, or process 
file requests for application servers, or manage and moni- 
tor the server components in the case of the System Moni- 
tor. 

Note: 

■ This routine must return immediately so that the Server State 
Manager can continue to operate in the main thread. 

■ This routine may make use of the Server Configuration Manager 
for obtaining specialized configuration information 

Input: None 

Output: Value of 0 if successful else error condition. 
Errors: TBD 



StopProcessing 



Description: Called by the Server State Manager. This routine must 

be defined by the specialized server component type. 

1 . Reverse all actions performed by the StartProcessing routine. All 

worker threads should be joined or pooled in waiting state. 
Successful return from this routine will put the server component into 
the IDLE state. 
Input: None. 

Output: Value of 0 if successful else error condition. 
Errors: TBD 



UpdateConfig 



Description: Called by the Server State Manager. This routine must 
be defined by the specific server component type. The purpose of this 
routine is apply dynamic configurations or update specialized configu- 
rations that are unique to this server component. 
<may require adding a new state to separate dynamic and static con- 
figurations> 
Input: None. 

Output: Value of 0 if successful else error condition. 
Errors: TBD 
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HandleError 



Description: Component defined error handling routine to handle errors 
such as timeouts, etc. 

This routine will need to handle a number of error cases as are possible 
by the specialized component. The error information is maintained with 
the ServerComponent class. 
Input: None. 

Output: Integer value designating a handled error or failure. If the error 
cannot be handled then it is fatal. 
Errors: TBD 



Run 



Description: This routine implements the main processing loop for the server 
component and runs in the main thread. This routine drives the server component 
by initiating state requests from the System Monitor. 
Note: The Server State Manager always runs in the main thread. 

1 . Call ProcessRequest to transition the server component into the initially re- 
quested state. 

2. Enter main processing loop 

a. Check for requests from the message service. 

b. Call ProcessRequest to service the request. 

c. Send acknowledgement for the request to the message service. Ac- 
knowledgement includes new state, load info, and status. 

Input: Initial Transition Request 

Output: None. This routine should never return 

Errors: None. 



Server Component Main Loop: 

The following main loop is common to all server components: 
void ServerComponent::Run(ServerRequest Request) 

{ 

ProcessRequest(Request) ; 
while (1) 

{ 

Request = Listener->GetRequest(); 
ProcessRequest(Request); 

Listener->AckRequest(Request, GetState, GetLoad, GetStatus); 

} 



^include "ServerArgs.h" 
^include "Server.h" 
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int main(int argc, char* argv[]) { 

Args = new ArgList(); 

Args->ProcessArgList(argv, argc); 

Server = new ServerComponent(GetValue(SID), 

GetValue(DSN), 
GetValue(DBUSER), 
GetValue(PASSWD)); 

Server->Run(START_PROCESSING); 

i 



Command Line Utilities: 



class NameValuePair 
{ 

private: 

char* Name; 
char* Value; 

public: 

NameValuePairO; 
~NameValuePair(); 
char* GetValue(void); 
char* GetName(void); 
char* SetName(char*); 
char*SetValue(char*); 

2i 



typedef int (*pFunc)(NameValuePair*); 

struct ArgTblEntry 
{ 

char* Name; 
bool Required; 
char* DefaultValue; 
pFunc ProcessFunction; 

ii 



ArgTblEntry const ServerArgsTbl[] 


= { 




{"sid", 


true, 


o, 


&ProcessSId}, 


{"dsn", 


true, 


o, 


&ProcessDsn}, 


{"dbuser", 


true, 


o, 


&ProcessDbUser} , 


{"dbpasswd", 


true, 


o, 


&ProcessDbPasswd} , 


{0, 

>; 


o, 


o, 


0} 



typedef vector<NameValuePair*> ArgVector; 
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class ArgList 

{ 

private: 

ArgVector 

const ArgTblEntry* 




ArgVec; 
ArgTbl; 


private: 

NameValuePair* 

char* 

char* 

int 

int 


ParseArg(char* Arg); 
ParseName(char* Arg); 
ParseValue(char* Arg); 
ProcessArg(NameValuePair*); 
FinalizeArgs(void); 


public: 

int 


ArgList(const ArgTblEntry*); 
ProcessArgList(char* argv[], int argc); 
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ProcessArgList 


Description: Process the entire argument list. 
In a loop for each argument argv[] . . . 

1 . Call ParseArg passing in argv[]. 

2. ParseArg passes the result to ProcessArg 

3. After processing the entire argument list and exiting the loop call 
FinalizeArgs 

Input: argv and argc as passed into main() entry point 
Output: integer value designating success or failure 
Error: 


ParseArg 


Description: Takes a char* argument and verifies that it follows that 
name/value syntax defined as <name>=<value> 
Input: Next char* argument on the list 

Output: NameValuePair. NULL will be returned in the event of a syntax 

error 

Error: 


ProcessArg 


Description: This routine performs the semantic analysis of an argument. 

1 . Look up name in the ArgTbl 

2. Verify that the value is valid 

3. Add the name value pair to a list of processed arguments called ArgVec 
list. 

4. If this name value pair already exists in the list then issue a diagnostic. 

5. Call the supplied processing function for this argument as specified in the 
ArgTbl 

Input: NameValuePair 

Output: Merger value designating success or failure (0 for success, positive 

integer for other errors) 

Error: 


ParseName 


Description: Verify that the Name part of the argument conforms to being 
alpha-numeric 

Input: char* Name part of argument 
Output: char* Name else NULL 
Error: None 


ParseValue 


Description: Verify that the Value part of the argument conforms to being 
alpha-numeric and/or punctuation characters 
Input: char* Value part of argument 
Output: char* Value else NULL 
Error: None 


FinalizeArgs 


Description: Post process the argument list. The purpose of this routine is to 
validate that all required arguments have been defined on the command line. 
Also processes and adds default arguments to the ArgVec. 
Input: None 

Output: Success or Failure 
Error: 



Configuration Manager: 
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class Tuple { 

string name; 
Value value; 

}; 

class Value { 

int type; 

}; 

class StringValue: public Value{ 
string value; 

}; 

class TupleValue: public Value { 
vector <tuple> tupleArray; 

}; 

typedef vector < tuple > ConfigArray; 

class ServerConfig { 
private: 

ConfigArray Array; 

public: 

ServerConfig(serverId, dsn, dbuser, dbpasswd); // Initialize from db 
ServerConfig(serverId, string filename); // To initialize from a file. 

ConfigArray* GetConfigArray(int serverld); 
Tuple* FindConfig(string Name); 
int Reload(void); 

Tuple* GetConfig(int serverld ,string Name); 

ii : 



ServerConfig 


Description: Constructor for Configuration Manager. 

1. Initializes configuration manager. 

2. Opens the database and gets configuration array 

Input: Server Id, Data Source Name, Database User name, and database 
users password. 
Output: None 
Errors: 


ServerConfig 


Description: Constructor for Configuration Manager. 

1 . Initializes Configuration Manager. 

2. Opens configuration file and reads configuration array. 
Input: filename of flat-file configuration. 

Output: None 
Errors: 
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GetConfigArray 



Description: Returns the entire configuration for a given server id. 
This routine always retrieves its information either from the flat file 
or the database. 

Input: Serverld specifying which server to retrieve configuration for 
Output: Returns a vector holding the configuration or NULL 
Errors: 



GetConfig 



Description: Returns the configuration for the specified name. This rou- 
tine always retrieves its information either from the flat file or the database. 
Input: Serverld specifying the server to retrieve configuration for and 
Name of configuration item. 

Output: Configuration Tuple. A Tuple may be a nested Tuple. NULL if 

an error is encountered. 

Errors: 



FindConfig 



Description: Returns the Tuple specified by the name. This routine does 
not go to the database or flat-file to get its value. Rather it finds the value 
in the ConfigArray maintained by the Configuration Manager. 
Input: Name of the configuration item. 

Output: Configuration Tuple. NULL if an error is encountered or the Tu- 
ple does not exist in the current configuration. 
Errors: 



Reload 



Description: Reloads the entire configuration from the database or flat- 
file. This routine may reload its configuration indirectly through the use of 
the System Monitor. In this case it will make a message request to the 
monitor and listen for the configuration results. 
Input: None 

Output: integer specifying success or failure. Zero will be returned in the 

case of Success. A negative value in case of error. 

Errors: 



Logging Utilities: 



class LogManager 

{ 

private: 

char* FileName; 
int MaxFileSize; 

char* ResourceFile; // message catalog file 
char* GetMessage(MsgNum, MsgStr) 

public: 

LogManager(ServerId,Size=10); 
LogMessage(MsgStr); 

LogMessage(ThreadId, MsgNum, MsgStr, ...); 
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LogMessage 



Description: Write message out to log file. There are two forms of 
LogMessage. The first will write out a message buffer as is (unformatted) 
bypassing the resource file. 

The second form will format the message. Both forms of LogMessage 
always pre-append a time stamp. 

1 . Lookup message number in the resource file and get message string 

2. format the log message using time stamp, thread id, etc. 

3. write out message into the log file. 

Input: Thread Id, Message Number, Message String, and variable num- 
ber of arguments. 
Output: None. 
Error: 



GetMessage 



Description: Routine returns a message string from the resource file for 

the message number specified. 

Input: Message number, C Locale text string. 

Output: Message string. Either way, Get Message will always pass a 
return a valid message string by either returning the string from the re- 
source file or by passing back the MsgStr passed in. 
Error: If an error occurs trying to get a message from the resource file, a 
message will be logged to the error log. 



class ErrorLog: protected LogManager 
{ 



private: 
public: 



1; 



LogLevel ErrorLogLevel; 

ErrorLog(ServerId, LogLevel=0, Size=10); 
LogError(ThreadId, ErrorNum, ErrorMsgStr, ...); 
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LogError 



Description: Writes output to error log file. 

1 . Check that the message level against the current ErrorLogLevel. 

2. Format the message and call the long form of LogMessage to write the 
buffer out to the file. 

Input: 

1 . Threadld: Thread identifier to help with the debugging process. 

ErrorNum: Error number used to uniquely identify an error message in 
the resource file. 

ErrorMsgStr: Message string which includes stdio like string formatting. 

variable list of arguments to be inserted into the message string per 
the format. 
Output: None. 
Error: 



Testing design 

Each of the components that make up the Server Component Framework will be able to 
be tested independently of the other components. Each component will have a main en- 
try point defined withm a testing .exe to accomplish the Unit testing phase. 

Testing of the component framework will be done in phases. Each of the phases is de- 
scribed below along with its dependencies. 



Phase 1: Unit testing 

Test basic components that make up the frame- 
work. Each components functionality, restric- 
tions, and boundary conditions will be tested. 

Will allow testing common configurations for a 
single server component. This round of unit test- 
ing will test the integrated component utilities 
and framework. 


Dependencies: 

1 . ServerComponent class 

2. ServerStateMgr class 

3. ArgList class 

4. Logging Utilities 

5. Configuration Manager (flat-file) 


Phase 2: Unit testing (full functionality) 
Test full functionality including messaging inter- 
faces and database connectivity. 


Dependencies: 

1. Phase 1 

2. Database connectivity 

3. Messaging Service 


Phase 3: Integration Testing 


Dependencies: 

1. Phase 2 

2. System Monitor (including 
backup) 

3. SLiM Server, App Server, Web- 
Server 


Phase 4: Stress Testing 

See section on stress testing for details 


Dependencies: 

1. Phase 3 
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Unit testing plans 

Command Line Utilities 

The Command line utilities will be tested in a stand-alone module called cmdline.exe. It 
will support the command line arguments defined in this document. 



Configuration Manager 

The configuration module is a stand-alone module which will be tested using a config- 
test.exe executable. The executable will exercise all of the interfaces described above. 
The configtest.exe executable should be testable in the DB and the non-DB mode. 



Logging Utilities 

The logging utility will be built as a DLL (otlog.dll). We will provide a binary otlog- 
test.exe which will exercise each of the interfaces mentioned above. 



Server State Manager 



The Server State Manager and the Server Component Framework will be tested inde- 
pendently of specialized components. The routines that require specialization (Start- 
Processing, StopProcessing, HandleError and UpdateConfig) will be provided to sim- 
ply return successfully. 

Stress testing plans 



Stress testing will require having at least the System Monitor functionality implemented 
since it is used to drive the server components. 

1 . Test to repeatedly start, stop, reconfigure the server component. 

2. Test to crash machines with server components to validate: 

a. data persistence. 

b. detection capabilities and response. 

c. auto restart. 

3. Test to kill individual server component processes. 

a. data persistence. 

b. detection capabilities and response. 

c. auto restart. 

4. Test lost database connectivity 

5. Test lost of messaging capabilities 

a. repeatedly losing and re-establishing messaging connectivity 

6. Test error recovery under adverse conditions. 

7. Test recovery from running out of memory, thread resources. 

8. Test recovery from threads dying. 

9. etc. 
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Coverage testing plans 

1 . Goal: 1 00% path flow coverage. Only exceptions for known error conditions that 
cannot be practically reached (e.g. thread synchronization, etc.) 

Cross-component testing plans 

The following pair-wise testing will be performed: 

1 . framework/database (phase 2) 

2. framework/messaging (phase 2) 

3. framework (System Monitor) /framework (backup Monitor) (phase 3) 

4. framework/Web Server (phase 3) 

5. framework (System Monitor) /framework (Other Servers) (phase 3) 

Upgrading/Supportability/Deployment design 

1 . Each error condition will be documented with explanations and practical work- 
arounds 

2. Component framework will support enhanced debug option to dump additional de- 
bugging information to special log files. 

Open Issues 
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Michael Beckmann 



Functionality 

The role of the System Monitor is to monitor the state of the Application Servers and 
SLiM servers within an eStream deployment. In addition, it also manages a back-up 
System Monitor. 

1 . The System Monitor provides the following key services: 

a. Monitors and reports server load across machines 

b. Monitors server state across machines. 

c. Acts as a communication conduit to the database for configuration 
information needed by the server components. 

d. Initiates state changes within the server. 

i. Start/Stop servers 

ii. Sends requests for servers to update their configurations 

2. The system monitor runs as it's own process. Within a multi-system deployment, 
there will be at least two monitoring processes, each on a different machine: 

a. One monitoring process will act as a primary and the others as backups. 

b. In the event that the primary monitor goes down, one of the backups will take 
over the primary monitoring responsibilities. 

3. The monitoring process can re-launch a server side process if a process terminates 
unexpectedly (this is a configurable option). 

4. The system monitor manages server state by maintaining regular communication via 
a heart beat protocol between itself, the backup monitors, and every logical server 

5. The system monitor's heart beat supports a light-weight messaging protocol between 
components to initiate state changes. 

a. Simple request pulse. 

b. Stop request pulse 

c. Configuration request pulse 

6. The monitor will raise an alarm if it does not receive a heart beat response from the 
servers within a specified period of time. 

7. The system monitor's heart beat request rate is a dynamically configurable parameter 
maintained within the database. 

a. The rate can be changed through the administrative UI. 
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System Monitor Component Overview: 



The System Monitor is made up of the following distinct components: 

1 . Server Component Framework (described in detail in the Server Component 
Framework Low Level Design) 

a. Server State Manager 
b* Configuration Manager 

c. Messaging Utility 

d. Logging Utility 

e. Command line Utilities 

f. Thread Management Utilities 

2. Load Monitor 

3 . Server Component Manager 

a. Remote Process Launching Utilities 

b. Heart-beat protocol 

4. Database Request Manager 

a. Database request protocol 

Server Component Framework: 

The System Monitor is extended from the Server Component Framework by providing an 
implementation for the following plug-able interfaces: StartProcessing, StopProcessing, 
UpdateConfig, and HandleError. Each of the interfaces is described in detail below. 
The routines StartServer and StopServer, which are also part of the interface are 
inherited from the Server Component Framework and do not need to be provided by the 
System Monitor. 

System Monitor State Transitions: 

Since the System Monitor is extended from the Server Component Framework, it needs 
to provide the four interface implementations called out above for each state transition. 
This section gives a brief overview of the primary state transitions and what is going on 
within the System Monitor for each transition. 

STOPPED -> IDLE: 

When the System Monitor is initially started by the Webserver it transitions from the 
STOPPED state to the IDLE state via a call to the StartServer routine. In this transition 
the following events occur: 

1 . Configuration manager reads common configuration from the database. 

2. Apply configuration and initialize (System Monitor) 

3. Change state to IDLE state. 

4. Send acknowledgment 
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In the IDLE state the System Monitor is still maintaining only its main thread which it 
inherits from the Server Component Framework and runs the Server State Manager, 
Configuration Manager, and the Messaging Service: 



IDLE->PROCESSING 

When the System Monitor transitions from the IDLE state into the PROCESSING state 
via a call to StartProcessing it creates a number of processing threads to do its work. In 
PROCESSING mode the System Monitor employs a "Boss-Worker" parallel 
programming model : 

1 . The System Monitor creates two processing threads that acts as the "boss" threads. 

2. The first thread is for the Server Component Manager to manage each and every 
server in the deployment. The thread entry point is MonitorServers 

3. The second boss thread is used for the Database Request Manager to service database 
requests from each of the server components for configuration and any special 
application server requests. The thread entry point is ProcessDbRequests 

4. The boss thread (Server Component Manager) reads the common configuration for 
all server components and spawns worker threads for each logical server component. 
A thread will be allocated to monitor each logical server processes. The thread entry 
point for each of the worker threads is ManageProcess 

5. The Load Monitor is initialized and runs as a service within the Server Component 
Manager boss thread . 

Below is an illustration of the thread model/architecture when the System Monitor is 
PROCESSING state. 
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Load Monitor: 

The Load Monitor's role is to aggregate the load information for each server component 
within the deployment and update the information to the database. 

The Load Monitor receives regular load information from each of the server component 
processes via the Server Component Manager. The load information is provided in the 
acknowledgement to regular heart-beat requests that each of the worker threads initiates 
with the server process that it is managing. 

The Load Monitor runs in the "boss" processing thread of the Server Component 
Manager. The Load Monitor provides thread safe interfaces. 

For each application being served within the deployment the Load Monitor maintains a 
list of servers and their response times. It periodically updates the database with these 
lists which are then consumed by the SLiM servers. The frequency at which it updates 
the database is configurable. 

Server Component Manager: 

The System Monitor manages each and every server component process within a 
deployment. This management activity is undertaken by the Server Component 
Manager. 
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The Server Component Manager spawns a worker thread for each server components 
within a deployment. The primary entry point for each of the worker threads is 
ManageProcess. This routine has the responsibility of starting and stopping each process 
and tracking its state, as well as requesting state changes in the component it is managing. 



Launching and Terminating Remote Processes: 

The System Monitor has the responsibility of launching server component processes 
across the deployment. The deployment can be made up of a heterogeneous set of 
servers. In order to accomplish this the system monitor communicates with either a Unix 
daemon or an NT service that runs on every server within the installation. 

The service/daemon is a very simple process that listens on its connection for requests to 
launch or terminate a process. 

If the daemon dies or is accidentally killed, the system will automatically re-launch the 
daemon. There are facilities for this on Unix and on NT. 

The System Monitor communicates with the service via the EMS utilities and a unique 
protocol defined below in the interfaces section. 

Once a process is launched the daemon/service does not track status or trap output of the 
process (fire and forget). Tracking of the process will be initiated by the System Monitor 
via the heart-beat protocol. 

Database Request Manager: 

In order to save database licensing costs, it has been decided that the Application Servers 
not have direct database connectivity. Instead, the Application Servers will depend on 
the System Monitor as an intermediary to read and write data from the system database. 

The System Monitor is used on behalf of all the other servers to interface with the 
database for retrieving configuration information and/or any other information that the 
Application servers may need. 

The Database Request Manager is launched from the StartProcessing interface. It is 
launched in its own boss thread. The thread entry point is ProcessDbRequests. 

The Database Request Manager also defines a protocol for each of the database interfaces 
defined in the WebServer/Database interface document. 

Data type definitions 

The System Monitor maintains a few key data structures beyond what is provided by the 
Server Component Framework: 
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Request Queue: 

The System Monitor makes use of mutex protected queues to communicate between 
threads. For example, each worker thread will maintain its own mutex protected input 
queue. The following components will maintain input queues: 

1 . Boss thread for Server Component Manager 

2. Boss thread for Database Request Manager 

3 . Each server monitor worker thread 

A queue entry models, fairly closely, the data contained within the messaging protocol 
between servers as defined in the Server Component Framework. This data can be used 
for the input queues for each worker thread managing individual work components. 

The details of the input queues themselves is defined in the common thread API. 



struct QueueEntry { 
int Request 
ServerlD SenderED 
ServerlD RecieverE) 
ServerlD TargetID 
Data 

} 



Managing Worker Threads: 

The System Monitor's boss thread (MonitorServers) maintains an in memory list of all 
known servers within a deployment, and their associated worker thread ID. In addition, it 
maintains information about request queues for each worker thread. This list is 
maintained as the ServerlnfoList The list is made up of entries which maintains all the 
necessary information to start/stop/manage/communicate/etc. with a server component. 
The ServerlnfoList is traversed by the Server Component Manager and the Load 
Monitor. 

struct ServerlnfoEntry 
{ 

ServerConfig* Config; //configuration for each server component 
Thread ThreadID; //Thread ID in which the Server Component 

//Manager is executing 
ThreadQueue* InputQueue; // processing queue for this thread (defined in thread 

// utilities package. 

}; 

typedef vector <ServerInfoEntry> ServerlnfoList; 
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Load Monitor: 

The Load Monitor maintains the following data structures: 
struct ServerLoadEntry 

{ 

ServcrlD 

Loadlnfo // still needs to be defined 

}; 

struct ServerSetEntry 

{ 

int AppID 

list <ServerLoadEntry> ServerSet // ordered list of application servers 

}; 

typedef vector <ServerSetEntry> LoadMonitorList; 

class LoadMonitor // thread safe/ re-entrant 

{ 

private: 

LoadMonitorList List; 

public: 

LoadMonitor(); 
~LoadMonitor0; 

UpdateLoad(ServerID, Loadlnfo); 

li _ 

Server Monitor: 



int MonitorServers(void); // non-member function used as thread entry point 
int ManageProcess(vector <ServerInfoEntry>); // non-member function used as entry 

// to for worker threads 



System Monitor Configurations: 



Configurations unique to System Monitors include: 



Information 


Supports 
Dynamic 
Config 


State 


Description 


LoadUpdateRate 


Yes 


All 


Frequency at which the Load Monitor will 
update the database with aggregate load 
information. Value is specified in 
milliseconds. 

If the value is zero then the Load Monitor 
will not update the database essentially 
disabling load monitoring. 
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Database Interface Protocol: 

There will be a unique operation code for each database interface. Likewise there will be 
an acknowledgement. 



OpCode 


senderlD 


receiverlD targetID 


Data 



Interface definitions 



Server Component Framework: 

The implementation of the System Monitor is derived from the generic ServerComponent 
class. Refer to the Server Component Framework Low Level Design. 

Implementing a SeverComponent requires the definition of the following methods: 



Description: This routine is called to start a server process and 
bring it to the IDLE state. It assumes that the server is in the 
STOPPED state. This routine performs all the necessary 
initialization and configuration common to all server components. 

This function is predefined in the ServerComponent base class. 
The System Monitor has no need to override it. 

Input: None. 

Output: Integer value designating return status. Success = 0. 
Errors: 



StartServer 



StopServer 



Description: The routine StopServer is called to cleanup and 
terminate a server process. It assumes that the server component is 
in its IDLE state and is therefore not processing any requests. 



This function is predefined in the ServerComponent base class. 
The System Monitor has no need to override it. 

Input: None. 

Output: Integer value designating return status. Success = 0. 
Errors: 
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StartProcessing 


Description: This routine initiates all the activities unique to the 
System Monitor: 

1 . Launches a boss thread. Call MonitorServers as the boss 
thread entry point. 

2. Launch another boss thread. Call ProcessDbRequests as the 
primary entry point. 

3. Return immediately 

Note: The StartProcessing routine must return immediately else 
the Server State Manager will be blocked. 
Input: None. 

Output: Integer value designating return status. Zero if success else 
error 

Errors: Error launching thread. 




StopProcessing 


Description: This routine will perform the following activities: 

1 . Terminates each server component's monitoring thread. This 
effectively disables the monitor from managing any executing 
server components including stopping existing components or 
starting new ones or servicing configuration requests. 

2. Terminates the Database Request Manager. 

Note: The primary system monitor will only execute this interface 
if a system administrator explicitly changes the monitors state or an 
error has occurred within the monitor. 
Input: None. 

Output: Integer value designating return status. 
Errors: 




UpdateConfig 


Description: This routine will perform configuration changes 

specific to system monitor functionality. 

Refer to the section on system monitor configurations. 

This routine can be executed from both the IDLE and 

PROCESSING states. 

Input: None. 

Output: Integer value designating success or failure. Zero for 

success. 

Errors: 



HandleError 



Description: This routine is called by the Server State Manager 
whenever an error is encountered. If this routine can handle the error then 
the original calling state will be maintained. If the error cannot be 
handled the process will transition to an ERROR state and terminate. 
Input: None 

Output: Integer value designating success or failure, zero for success. 
Errors: Error specifying that the error could not be handled. 
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Server Monitor: 



MonitorServers 



Description: This routine is defined as a non-member function and is 
the thread entry point for the "boss" thread of the Server Component 
Manager. It is the primary thread for managing and monitoring all the 
server components. 

1 . Create an input request queue for this thread 

2. Start Load Monitor 

3. Go to the database and get a list of all the server components 
calling GetServers as defined in the Database Low Level design 
document. 

4. For each server component retrieve its common configurations 
calling GetServerConfig and create and entry into the 
ServerlnfoList 

5. Spawn a worker thread to manage/monitor the backup System 
Monitor. Call ManageProcess as the thread entry point passing 
in the ServerlnfoEntry After the backup monitor has been 
launched, repeat step 4 for each server component. 

6. Check the input queue for new worker requests. 

7. Loop back and redo steps 3-6 looking for new servers to manage 
Input: None. 

Output: 
Errors: 
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ManageProcess 


Description: Non-member function that launches and monitors thp 




specified server component processes. It is expected to be the start 




routine for its own thread. This routine is called from 




MonitorServers. Performs the following steps: 




1. 
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uuuii^ui duuu iJiiuiiiidLion proviueu in me oervcriuioit-niry. 




2. 


C /nen a noint-to-noint mPQQj»oiTi(T rnnnprtinn ir\ tH^ -np»vi/lA/ 
wpvai a pwmi iu puilll JJIC/C>:><l£lllg l/luillCl'lIlMl IU UJC JJCWly 






1 ?11 in phpn QprA/PT cr\mr\r\r\f>rti 




3. 


uiiuaicr a i^^uiai iicaj i-ucdi icqucM <ii iiic idie ueiineci wiinin me 






confi cmratinn 

v/ Willi ^1*1 CllUJlJ. 




4. 


Pntpr monitorino Innn 

-LdllVJ 111VJ1J1 lv/1 Jllti lUVJ LI 






a. Check for requests from the input queue for this server 






component. 






b. Send the request to the server currently being monitored 






by this thread. 






c. Listen and wait for response; timeout if wait is too long. 






d. If a response is received update state information in 






ServerlnfoEntry and update the database as necessary 






calling SetServerState and SetServerLog. 






e. Enqueue load information calling UpdateLoad 






f repeat a-f 




Input: ServerlnfoEntrv 




Output: Status of the server component on exit 




Errors; 




1. 


launch error 




2. 


messaging error 




3. 


unexpected server component termination 



Database Request Manager: 



ProcessDbRequ es ts 


Description: This routine is the thread entry point for the 




Database Request Manager. It is called from StartProcessing. 




1 . Connect to Database 




2. Create a listener using EMS utility 




3. Sleep until a request comes in and then enqueue the request. 




4. Worker threads will then dequeue and satisfy the db requests 




and terminate 




Input: db connectivity info 




Output: 




Errors: 
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Load Monitor: 



The Load Monitor maintains three public interfaces: 



UpdateLoad 


Description: This routine updates, in memory, server sets 
with the current load information. This routine is thread 
safe/re-entrant 

This routine is called by ManageProcess 

1 . Traverse server lists by application. If the application. If 
the application entry is not found create it and add it to the 
list. 

2. If the application entry is found search for the server entry 

3 . Update load information 

4. Call FlushLoadData per the database access frequency 
configuration. 

Input: Load information, Serverld 

Output: Integer value specifying Success or Failure. Zero if 
success. 

Errors: Error processing load information 


LoadMonitor 


Description: Initialization routine for the Load Monitor. 

Called by MonitorServers 

1 . Initialize server lists to NULL. 

input. iNone. 

Output: None. 

Errors: 


-LoadMonitor 


Description: Destructor for the LoadMonitor. 

1 . For each server flush all load data to the database calling 
FlushLoadData 

2. Cleanup server set data structures and exit. 
Input: None. 

Output: None 
Errors: 
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Primary and Backup System Monitor: 



The backup System Monitor needs to be able to take over the primary system monitor 
responsibilities in the event that the backup loses communication with the primary. The 
following interface will cause the backup monitor to take over as primary. 



SwitchPrimaryMonitor 



Description: Take over primary monitor responsibilities. 
This routine is called from the HandleError routine defined 
for a backup monitor. If the backup monitor doesn't hear 
from the primary monitor, it assumes that the primary monitor 
has died. 

1 . Update database validating monitor switch. 

2. If no database connectivity then go back to listening mode 
until the next time out. 

3. If database connectivity then attempt to shut down the old 
primary System Monitor just in case it is still running. 

4. Go through the same steps as the primary monitor would 
do when its StartProcessing routine is called including 
starting up a new backup system monitor 

Input: None 

Output: integer value designating success or failure 
Error: 
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Launching and Terminating Server Processes: 



Termin ateRemoteProcess 



Description: This routine terminates a remote process 
either on a local machine or on a remote machine. This 
routine is called by ManageProcess. 

1 . Verifies that the target machine is up and running 

2. Send message to daemon/service to terminate the 
process. Uses EMS messaging service. 

Input: Process info, target machine 

Output: integer return specifying success or failure. 

Errors: 



StartRemoteProcess 



Description: This routine spawns the requested process either on 
the local machine or on a remote machine. This routine is called 
by ManageProcess 

1 . Verifies that the target machine is up and running by sending 
ping to daemon/service on the specified machine. 

2. Send message to daemon/service to launch a process passing 
the process path and any arguments. 

Input: Process name, target machine, attributes 

Output: integer return where 0 designates success else error code 

is returned. If successful the routine will also return process info 

Errors: 

1 . machine not responding 

2. executable not found 

3. failure on launch 



Daemon/Service 


Description: Unix daemon/NT service which runs on every server 




machine within the deployment. Its role is to dispatch requests to 




create and terminate processes, obtain process information. 




1 . Initialize/Create a listener using EMS 




2. Go to sleep until a request comes in 




3. Dispatch request 




4. Return information to requestor 




5. Perform any necessary logging 




6. Repeat 2-6 




Input: listening port 




Output: None 




Errors: Errors will be written to a special log file 



Testing design 
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The System Monitor will provide a command line option to facilitate testing the 
component in its various testing phases. The option will allow the system to manage its 
interfaces through flat files. 

Unit testing plans 



To unit test the System Monitor it will require a single generic server component running 
on the same machine. 



Phase 


Description 


Dependencies 


Phase I: Local process 
management 


Cycle through all the state 
changes for a single local 
generic server component 


1 . Functionally complete 
Common Server 
Framework 

2. Generic Server 
component 


Phase II: Remote process 
management 


Same as Phase I for remote 
processes 


1 . Phase 1 dependencies 

2. Server daemons 


Phase III: backup monitor 
failover 


Verify that the backup 
monitor fails over and 
continues to manage the 
system 


1 . Phase II dependencies 

2. Backup Monitor Server 



Stress testing plans 

Tests should include: 

1 . Max # of generic server components on a single machine. 

2. Max # of generic components across multiple machines. 

3. Max # of generic servers changing state at least once per second 

4. Test to repeatedly start, stop, reconfigure each server component. 

5. Test to crash machines with server components to validate: 

a. data persistence. 

b. detection capabilities and response. 

c. Auto Restart. 

6. Test to kill individual server component processes. 

a. data persistence. 

b. detection capabilities and response. 

c. Auto Restart. 

7. Test lost database connectivity 

8. Test lost of messaging capabilities 

a. repeatedly losing and re-establishing messaging connectivity 

9. Test error recovery under adverse conditions. 

10. Test recovery from running out of memory, thread resources. 
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1 1 . Test recovery from threads dying. 



Coverage testing plans 

The System Monitor will achieve 100% code coverage with the exception of error 
conditions which are possible however difficult to reach in practice. 

Cross-component testing plans 

The System Monitor interacts with the following components: 

1 . System Monitor/Database 

2. System Monitor/WebServer 

3. System Monitor/Server Component Framework (other servers) 

4. System Monitor/Backup Monitor 

Upgrading/Supportability/Deployment design 

1 . All diagnostics will be documented as to their root cause and workarounds/actions to 
be taken. 

2. The system monitor will support an enhanced debug support which dumps additional 
information to special debug logs. 

Open Issues 

1 . Need to identify a mechanism to ensure that we do not have more than one primary 
monitor running at any given time. 
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eStream Messaging Service Low Level Design 



Version J.J 
Sameer Panwar 



Functionality 

Estream is fundamentally a networked system, and thus must rely on a communication 
infrastructure as part of its foundation. The EMS subsystem provides the API for 
eStream processes to communicate with remote eStream processes. It is subdivided into 
layers and is intended to freely support different combinations of choices from each layer 
in an extensible manner. E.g. XDR over HTTP over SSL, while new layers can be added 
in the future (for compression, custom encryption, etc.). This is important considering it 
will be used across the variety of eStream components, including the client cache man- 
ager and all the servers. 

The layers are enumerated as follows: 

1 . Packaging Utility [XDR/SOAP/name-Value pairs, etc.] 

2. Procedure Call Layer (PCL) 

3. User Layer(s) 

4. Message Transport Layer (MTL) [SSL+TCP/HTTP+TCP/MSG+TCP] 

The Packaging Utility is primarily used to pack and unpack the parameters of the mes- 
sage into the body. It isn't really a true layer, in that it is visible to the other layers— they 
also use it to pack their data into their headers (except if the layer is a given standard). 
Thus all layers are subject to the same packaging scheme, unless they provide their own 
internally. The various packaging utilities won't be tied to EMS directly— they will be 
available for packaging data into any buffer. 

The Procedure Call layer provides the functionality to assign API and Function identifiers 
in order to define a remote API, as well as providing a dispatch mechanism on the server 
side to enable appropriate processing of incoming messages. This layer will also handle 
callbacks for replies to asynchronous messages. The implementation for eStream 1 .0 for 
this layer will be called EPC for eStream Procedure Call. 

The User layer is present to allow the user of EMS to add custom headers on the send 
side and perform early checks (e.g. on AccessTokens) on the receive side before being 
queued for service in the thread pool. This could also be used for compression or a cus- 
tom security layer, and multiple User layers could be put in place. 

The Message Transport Layer is special in that it is this layer that uses the actual socket 
interface. However, it has the additional functionality (referred to as MSG above) to rec- 
ognize message boundaries via a magic number and a message size (which is also done 
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via an HTTP header or an SSL header). This enables us to grab a complete message over 
the stream-oriented TCP before sending it up to higher layers. Note that UDP might not 
be supportable in EMS— there may be inherent assumptions that the underlying network 
protocol is reliable and connection oriented, but I won't know for sure until EMS is im- 
plemented for TCP whether it will be adaptable to UDP. Also, the MTL is also responsi- 
ble for contacting proxy servers in the manner appropriate to the protocol they're using 
(SSL or HTTP). 

Messages are linked lists of memory blocks (which also track size used), each block be- 
ing a layer header, with the original message body last. They are always flattened in 
MTL before sending, but can be flattened elsewhere if desired. Probably we'll use 4k 
blocks for each header, and a 64k block for the body. This linked list of blocks is called 
the Message Buffer, which is contained in the Message object. Messages also contain 
per-message info relevant to the various layers, including API & FUNC #'s, Message 
IDs, etc. This per-message info is used to generate the appropriate headers for outgoing 
messages, and is unpacked into the Message for incoming messages. It is the sum of all 
per-message info any layer might want, placed in the single Message class, even though 
obviously only the layers in use for the given connection will actually be active. 

EMS layers have: 

Header data members 
BuildHeader function 

BodyXform transformation function (e.g. compression, but often just null) 
Send function 

Internal Connection info (nonces, request ids, SSLCTX, etc.) 

Reset function (when need to re-establish a connection) 

Session info that's global for the layer, static data (session ids, callbacks, etc.) 

In usage, each layer, in the outgoing direction 

1 . Gets outgoing Message, becomes owner of the memory. 

2. Builds a header (needs to grab a free block) using per-message info set up by the 
user in the Message, and adds it to the Message Buffer. 

3. Applies any applicable transformation to the message body. 

4. Passes the Message it to the lower layer. 

5. Each layer is NOT to remember anything on a per message basis, since it won't 
be notified if the message is lost! 

Each layer, in the incoming direction 

1 . Gets the incoming Message. 

2. Unpacks the header, performs any relevant checks and transformations on the 
message body (that is, the body relative to this layer's header). Increments a 
pointer so that a higher layer can know where its header starts. 

3. Places any relevant header info into the per-message info in the Message, e.g. 
Message IDs. This layer-related info will thus be available outside EMS. Only a 
limited number of these will actually get set, since some of these are only relevant 
in the outgoing direction. 
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eStream Messaging Service Low Level Design 

4. Passes the Message (or an error) to the higher layer. If the higher layer is the 
EMS caller, then the Message is delivered to them so that they can then unpack 
their data. 

Steps to use EMS: 

1 . First, a Connection object must be initialized, describing the remote server, the 
stack of layers and packaging utility, along with other configuration. 

2. Next, the user must package their data into a Message Buffer. 

3. Then they must set up any required per-message info in the Message object. 

4. Finally, they can send the Message (this triggers the actual layer processing, and 
then the Message gets queued to go out the network). 

Message Stack 

A Message Stack is an object that describes the order of the actual layers being used for a 
given remote server, and also binds to a packaging utility, which is used by the layers and 
user to marshal and unmarshal data. The Message Stack doesn't actually contain the 
layer objects themselves, just pointers, so there is just one Message Stack object. For any 
listening port on a server, only one Message Stack is used, and essentially defines the 
language spoken on that port. Other ports on the same server can use different Message 
Stacks. 

Connection 

The Connection object contains all relevant data EMS requires to manage a connection 
with a remove server. It contains the relevant layer objects, which keep track of connec- 
tion-level info, as well as a Message Stack. It also contains the remote server's DNS 
name and/or DP address and port number, and the same for a proxy server if necessary, 
these being used by the Message Transfer Layer. The reply timeout and socket handle 
and state are also maintained here. There is one Connection object for every combination 
of layers (and packaging utility) and remote server we care to use, and instantiates each 
layer object and initializes it (and assembles the Message Stack) as part of its constructor. 
A given connection will be configured to support synchronous or asynchronous commu- 
nications, but not both. A Connection also contains an outgoing message queue, and a 
(pointer to a) buffer for messages in the process of being read or written to the network 
along with a current position index. 

Connection Management 

I/O can be nonblocking, but the first 3 layers are not allowed to block on an OS call, so 
stuff only gets queued in the MTL. Thus there is one well defined place where 
sends/receives of messages can be resumed, and the states involved are described below. 
If the socket is blocking, then we need to set a timeout and return an error on a failure. 

A special thread called the socket watcher does a select on active sockets, and triggers 
actions based on the socket's connection state and message state. If the state isn't CON- 
NECTED, then it continues the relevant handshake. If it is CONNECTED, then it sends 
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the outgoing message, or dispatches an incoming one. It is also responsible for handling 
timeouts. 

Connection states: 

DISCONNECTED 

TCP_HANDSHAKE_WAIT select(write/except) 

MTLHANDSHAKERWAIT select(read) 

MTL_HANDSHAKE_W_WAIT select(write) 

CONNECTED select(read) 

Message state (outgoing) for each connection: 

EMPTY select(read) 
MSGREADY select(read/write) 
MSGJSENDING select(read/write) 

Relationships: 

1 Connection Object per remote server, and 1 associated Message Stack for the Connec- 
tion Object. Thus, if communication to the same server is required using two different 
Message Stacks, then 2 Connection Objects are required. Multiple Connection Objects 
with the same Message Stack to the same server are allowed for clients, but discouraged, 
since that eats up server resources. 

Handshakes 

Some of the underlying protocols need to perform handshakes before they can exchange 
user data. E.g. SSL needs to perform authentication and key exchange and obtain a ses- 
sion before the user's data can be encrypted for transmission. These handshakes can take 
more than one exchange to establish the session, e.g. SSL takes 2 l A round trips (on top of 
TCP's 1 Vz round trips). SSL's interface for handshaking supports nonblocking I/O by 
implementing an internal state machine to handle the handshake protocol. EMS will ex- 
port this interface on the send side (by keeping a connection state); for receives, EMS 
will have to check to see what the connection is waiting on (read or write, even if the user 
request was different) and check for it during a select(). If data is returned when the con- 
nection is still being set up, then the handshake is continued to read the waiting data. Af- 
ter the handshake is complete, the layer must check for any pending message, process it 
with the session data, and pass it down. While a handshake is going on, all new requests 
are processed down to the MTL and then queued. This means that header data members 
relating to the MTL cannot be changed, so access to them will be through functions 
which will fail on a call for a given connection object that is not disconnected. Hand- 
shakes are NOT supported at any higher layer than the MTL, since that would entail too 
much complexity. Thus any other desired handshaking must be implemented on top of 
EMS, though handshaking could be possibly supported within higher layers if communi- 
cation were only synchronous, if the need arises. A successful SSL handshake will set 
the session ID. If the handshake fails on trying to re-use an old session ID that expired, 
then the session ID is cleared, and a full handshake is required again. 

Global EMS Stuff 
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EMS at the global level manages the messages being sent. When timeouts are being 
used, EMS can assume that for every outgoing message some response message must be 
returned from the server, and will report a timeout to the caller if a response is not re- 
ceived by the deadline. EMS requires a Message ID (also used by the PCL dispatcher) 
to be passed in by the user, which it stamps on the outgoing message, and is present on 
the reply message which cancels the timeout. Message IDs are also used by the PCL dis- 
patcher, so that the client callback can match the response with its request. EMS requires 
that the user use a monotonically increasing Message ID, and will thus reject any reuse 
(this enforces uniqueness of Message IDs for the lifetime of the process). Since Message 
IDs are low-level EMS details, they live in the MTL. EMS keeps an ordered queue of 
Message IDs and timeouts (but throws away the actual message contents once it's sent), 
and the socket watcher thread uses the earliest timeout in its selectO call. 

Asynchronous calls 

For asynchronous calls, client callbacks are registered one for each remote function 
(these are managed by the PCL), as well as a timeout function (which is handled by the 
MTL). The socket watcher thread gets messages as they come in, and the PCL will 
queue them for a dispatcher thread to process. 

Buffer/Memory Management 

Since we need to manage buffers efficiently, and also avoid possible attacks if we support 
very large message sizes, we should pick some reasonable maximum size for any mes- 
sage. For now, I will set that to be 64k (remember, this is the max size after packaging), 
which really may only affect the app server. This maximum size, of course, includes 
headers in any of our layers, and may be restricted even more when SSL is in use (due to 
its own restrictions). We should also consider the lifetime of memory buffers. In time 
sequence, normally what happens is that the user grabs a buffer for the main message 
body, packs their data into it, then passes it to the top EMS layer. Each EMS layer then 
grabs a buffer for the header, and at the last step this is flattened into yet another buffer 
(which frees the previous buffers), which is then passed to the network, after which the 
Message object can be freed. When a message is received, it is placed into a buffer in a 
Message object, which is given to the user, who is then responsible for freeing the Mes- 
sage object when they're done with it. Before that, the unpackaging functions may allo- 
cate memory for variable sized objects, which then also become the user's responsibility 
to free. 

Since the lifetime of many of the memory buffers is short, but the number of outstanding 
messages is generally pretty low, reuse of buffers can definitely help efficiency. For the 
various transient classes, I expect to implement a memory manager class that will keep 
the objects around instead of freeing the memory, and then reuse them when a new one is 
needed. Thus each object will have its own pool. Buffers will be managed the same 
way, probably coming in 4k (for headers) and 64k sizes. However, I'm not sure if I want 
to use this approach for memory allocated for the variable sized objects in the unpackag- 
ing functions; maybe I'll just use malloc for that. 

Socket Watcher 
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The socket watcher thread is required to support asynchronous messaging, by grabbing 
and dispatching messages as they come in, also making it the logical place to handle 
timeouts for asynchronous threads. This special thread needs two data structures— the 
list of sockets to watch and a queue of timeouts. Connections that are configured as syn- 
chronous never pass their sockets to the socket watcher. For asynchronous connections, 
whenever the connection state changes, the socket may be added/updated in the socket 
watcher's list (we need to know whether to check if the socket can be read from or maybe 
written to, depending on the connection state). When the MTL discovers that the TCP 
connection was closed, it removes it from the socket watcher's list. Note that the socket 
watcher is actually "just below" the MTL: it doesn't know how to decode a Message ID. 
When a socket is ready for reading, it gets the associated Connection object, and then 
calls the MTL (in the socket watcher's context) to pull the message out and then cancel 
the pending timeout. Then, still in the socket watcher thread, the message is pushed up 
the Message Stack until it is dispatched to another thread for actual servicing (as in the 
PCL). Then the socket watcher goes on to process any other ready sockets. Note that 
thus the socket watcher could process messages going to different MTL's. There is only 
a single socket watcher thread in EMS for a given process. The socket watcher also 
checks for incoming connections on the listening ports (there can be more than one on a 
server). 



The socket watcher is also responsible for completing sends of messages that were left 
incomplete (i.e. would have blocked) or are queued. A connection's message state re- 
flects this and has the watcher check for socket writability— when that happens, it then 
tries to complete the send as appropriate. In fact, if asynchronous messaging is being 
used, all actual writes over the network happen in the socket watcher thread, the user 
threads just dumping the messages into the queue and changing the connection state ap- 
propriately. 

When a user uses synchronous messaging, this is different in that the user's own thread 
then sends the outgoing message, waiting until the send is complete, and then does a se- 
lect on just that one socket with its own timeout. When the reply is received, then the 
MTL and higher layers are executed in the user's context, and no dispatch to another 
thread occurs (since the Connection is configured as synchronous). 

Clients vs. Servers 

Client and Server behavior is distinguished by the EMS calls they use. Clients use 
EMSMsgSend(. . .) which takes a Message ID and a timeout value, and if they're making 
asynch calls, then they must also set up callbacks. Servers use EMSMsgReply(. . .) which 
returns the above Message ID and has no timeout. Despite this difference, the socket 
watcher behaves the same on the client and the server. It simply does a select on all open 
connections and tracks timeouts. Of course, the server just disables the timeouts. Also, 
the server listens on a specific socket for incoming connections, while the client does not. 
However, servers have a different kind of timeout— the connection idle timeout, which 
means if a connection isn't used by a client for a certain interval, the server will drop the 
connection to conserve resources (sockets). The server's MTL will establish its own 
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callback with the socket watcher in order to close the connection when it times out (and it 
will also cancel timeouts whenever it receives a complete Message). The client's MTL 
will transparently detect the dropped connection and reestablish it when the next outgoing 
message is queued. 

Another difference on the receive side occurs in the PCL. There, the client side PCL will 
look at the PCL function number and dispatch to a callback (if the call was asynchro- 
nous), or pass it up to the user directly (since the watcher thread wasn't involved). On 
the server, only the socket watcher will get messages, and the PCL will dispatch the mes- 
sages to the functions that were registered for the corresponding PCL function number. 
Hmm.. These are actually similar enough that maybe they could be done with the same 
mechanism. Then the main difference between the client and server with respect to EMS 
is just that the server is listening for incoming connections and uses different timeouts. 

Errors 

I can't really describe this in detail now. This will have to come out of the implementa- 
tion. 

In general, if an incoming message is bad in some way, it'll just get dropped. If we care, 
we can keep a counter that is incremented for each bad message and raise an alarm if it 
reaches a threshold. . . 

Multithreading/Synchronization 

EMS will have 2 threads internal to itself, both of which are only required when asyn- 
chronous messages are configured or EMS is being used in a server process. The first 
thread is the socket watcher, which is started by the EMS initialization. The second 
thread is the PCL dispatcher, which is started by the PCL initialization (this thread is the 
context in which actual requests are processed and work is done, and this could in fact be 
a pool of threads with a queue in front). Other layers could add their own threads if de- 
sired. All other EMS functions are fully executed in the context of the caller. 
What are the shared data structures among threads? Since EMS functions could be called 
by different non-EMS threads, there's a good number of data structures that need to be 
synchronized. 

First, since multiple messages may be processed at the same time, the layers above the 
MTL need to lock any shared data that they manipulate internally (such as IDs or non- 
ces). Also, access to the memory buffer pools must be synchronized. Once the MTL is 
reached and the message is ready to be sent through the MTL, a lock is acquired (e.g. to 
protect the SSL_CTX structure) and only released when the bottom-level SSL or TCP 
interface returns, so only one thread can be blocked on a recv for a given socket at any 
time. The Connection's message queue and connection state must also be synchronized, 
as well as the socket watcher's timeout queue and active socket list. For connection-level 
stuff, it may be easier to lock the entire connection object (i.e. the whole time from the 
message body being passed to the top layer and the point where the assembled message is 
placed on the outgoing queue, thus covering all layers at once); this will probably be im- 
plemented first, and we can use finer grained locking if this is a performance issue. 
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General policies 

EMS does not ensure any ordering of processing of requests. Requests of course go out 
in the order they were sent, but the server may process them in a different order due to 
thread concurrency. If ordering is desired, it needs to be implemented in a User layer via 
some kind of queuing mechanism. 

HTTP 

[mostly taken from Bhaven's HTTP document] 

To make HTTP proxies happy, we'll implement a simpleminded subset of HTTP headers. 
For the request message, we'll use the format: • 

POST/HTTP1.1 

Host: <servername> 

Connection: KeepAlive 

Content-Type: octet-stream 

Content-length: <content_length> 

<message body> 

For the reply message, we'll use the format: 
HTTP/1.1 200 OK 
Content-Type: octet-stream 
Content-Length: <content_length> 

<message body> 

The HTTP+TCP MTL in EMS will also support a GET-type request with a URL but no 
message body, if the appropriate Message options are selected and the user provides the 
URL string, which they (or some higher layer) will have to package. The server-side 
HTTP+TCP MTL won't decode the URL, it'll just store it in the Message, and leave it to 
higher layers or the user to interpret it. This layer will assume the HTTP header will have 
a max size of 4kB, and report an error if it pulls more than out of the socket before reach- 
ing the body. 

Also, this layer may need to emulate HTTP 1 .0 behavior (i.e. no persistent connections) if 
the proxy isn't 1.1-friendly, but I'll leave that to be implemented later if necessary, and 
for now we'll notify our customers that they need a 1 . 1 -compliant proxy server. From 
what we've gathered so far, most deployed proxies should be 1.1-friendly. 

EStream 1.0 

The current plan for eStream 1.0 is to implement the following layers: 

Packaging Utility - adapt the Sun XDR routines. We'll add more for the clients Web 
server interface or Web server<~^SLiM interface later if required. 
Procedure Call Layer - we'll just implement one, since only one is required and all com- 
munications we have planned will use it. 
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User Layer - unknown, but we could implement AccessToken checks at this layer, before 

requests are queued at dispatch time. 

MTL - we will support SSL on TCP and HTTP on TCP. 

Synchronous messaging could be implemented first, but pretty much all of the asynchro- 
nous messaging needs to be in place for server functionality. Therefore release 1 .0 will 
have full asynchronous messaging support. 

Data structures/Interfaces 

First things first. EMS needs to be initialized on process startup. 

EMSInitialize(???) - this function sets up any global stuff required. 

class EMSBlock 
{ 

char * buf; 

uint32 bufsize; // size of the alloc'd mem that buf points to, for bounds checking 
uint32 size; // amount of buf that is occupied, i.e. pointer to where to add data. 

} 

class EMSMessage 

{ 

public: 

// actual message contents 
List<EMSBlock> MessageBuf; 

// PCL stuff for this message 
uint32 apiNum; 
uint32 funcNum; 

// MTL stuff for this message 

uint32 messageld 

(SSL related stuff perhaps) 

// HTTP stuff for this message 
boolean isRequest; 
char * URLString; 

//other stuff 

uint32 position; // index into MessageBuf where next layer should start reading 
// or writing. Also used to indicate where more data should be 
// read or written to from the network if the Msg is incomplete. 

flattenO; // this function flattens MessageBuf into just one block 

} 
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class EMSConnectionBase 
{ 

public: 

(functions to set some of the below values) 
SendMsg(EMSMessage * msg); // passes msg through the stack 
EMSMessage * GetMsg(); 

Reset(); /* disconnects, resets all state (clears Q's), can then be reused for other 
* servers with the same MessageStack as the given Connection object */ 

private: 

char * destName; 
addr destlpAddr; 
uintl6 port; 
char * proxyName; 
addr proxylpAddr; 

boolean isClient; // tells EMS how to behave, esp. wrt the timeout 

boolean synchronous; // actually only relevant if is a client 

uint32 timeout; // in milliseconds 

EMSMessageStack stack; 

EMSSocket * sockPtr; 

EMSConnState state; 

Queue<EMSMessagePtr> outgoingQ; 

EMSMsgState msgState; 

EMSMessage * sendingMessage; 

EMSMessage * receivingMessage; 

} 

class EMSConnection_XDRJSPC_SSLTCP : EMSConnectionBase 
{ 

private: 

EMSPackagerXDR xdr; 

EMSLayerEPC epc; /* for estream procedure call ? */ 
EMSLayerSSLTCP ssltcp; 

EMSConnection_XDR_EPC_SSLTCP (init stuff for XDR, EPC, SSLTCP layers 
as well as dest'n name & IP address, port, timeout) 

the constructor grabs the params to initialize the relevant layers, and 
places the layers in the proper order in the message stack. 

} 

} 

class EMSConnectionMgr 
{ 

[This class is used to get connection objects of a given message stack, 
by the socket watcher; actually must pass derived classes in as this type, 
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i.e. the socket watcher wants a "EMSConnectionMgr", but we give it a 
EMSConnectionMgr_XDR_EPC_SSLTCP, and the right stuff happens 
via the virtual functions.] 



class EMSPackagerBase [just an abstract base class] 
public: 

PutUint32(EMSBlock *, uint32) - 0; 
PutUint8(EMSBlock *, uint8) = 0; 
PutString(EMSBlock *, char *) = 0; 
[Etc., one function for every basic datatype supported. 
For custom datatype/structs, should build a routine 
for the struct that takes a packager object as an arg 
and uses its basic datatype packaging routines.] 

class EMSPackagerXDR 
{ 

[actually implements the above virtual functions using the XDR spec] 

[for outside EMS, the XDR functions will be visible as, e.g., EMSPutUint32XDR(char * 
buf, int * pos, uint32), where pos points to the place in the buf to place the uint32 and it 
gets incremented as appropriate, but these functions won't check bounds for buf.] 

class EMSLayerBase [another abstract base class! 
{ 

public: 

ProcessMsglnfEMSMessage *) = 0; [processes an incoming message] 
ProcessMsgOut(EMSMessage *) = 0; [processes an outgoing message] 
ResetQj 

private: 

int BuildHeader() = 0; 
int BodyXformO = 0; 

} 

class EMSLayerEPC : EMSLayerBase 
public: 

EMSLayerEPCfJ 
{ 

[starts dispatcher thread] 
GlobalRegisterAPI(API #, # funcs, func table); 



Omnishift Technologies, Inc. 



11 



Company Confidential 



eStream Messaging Service Low Level Design 



private: 

[table of pointers to function callback tables (one table per API), 
this table being statically allocated would mean that I'd set a max 
supported API number to be something like 256. This table is 
static data, i.e. visible and shared by all instantiations of this class.] 

} 

class EMSLayerSSLTCP 
{ 

public: 

EMSLayerSSLTCP(SSL params, TCP params) 
{...} 

private: 

uint32 curMessagelD; 
SSL_CTX * context; 

} 

class EMSMessageStack 
{ 

List<EMSLayer *> layerlist; 
EMSPackager * packer; 

} 

[this is really just used internally by the Connection object, of little interest anywhere 
else] 

EMSTimeout 
{ 

time deadline; 
uint32 messagelD; 
EMSConnection * conn; 

} 

EMSSocket 
{ 

public: 

ChangeState(EMSSocketState newState) 

[this modifies the FDJJETs below, as do the constructor & destructor.] 

private: 

socket sock; 

EMSSocketState state; // derived from connection state & message state 
EMSConnection * // for other open sockets 

EMSConnectionMgr * // for listen sockets 

static FD^SET readset; // used by selectQ 
static FD_SETwriteset; 
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static FD SET exceptset; 

} 

EMSRegisterStack(EMSConnectionMgr *, uintl6 listenjport) 
{ 

[tells socket watcher to bind this port with the given connection queue. 
Grabs a free EMSConnection for every connection made with a client 
and associates it with the new socket] 

} 

Most of these interfaces appear because of EMS' s layered structure, to support extensibil- 
ity, but are not meant to be used by the EMS user, and involve lots of lower-level grungy 
details that are transparent above EMS. In general those coding to use EMS only need to 
worry about the following: 

First, configure global data of layers being used, e.g. 
EMSLayerEPC: :GlobalRegisterAPI( . . . ) 
EMSLayerSSLTCP: :DoCertificateStuff( . . . ) 

Then, build a connection object appropriate to the server & API you want to use, e.g. 

foo = new EMSConnection_XDR_EPC_SSL_TCP( <c appserver.foo.com", &address, 
4567, 1 000 [timeout in ms], CLIENT, SYNCHRONOUS); 

Then create a message object and start dumping your parameters into it: 

msg = new EMSMessage; 

msg->apiNum = EPC_APIJXffiNT_APP_SERVER; 
msg->funcNum - EPC FUNC GET PAGE; 
msg->messageld = msgid++; 

EMSPutAccessToken(foo, msg, &accessToken) 
EMSPutUint32(foo, msg, appld); 

/* this is really an inline function that does foo->stack->packer->PutUint32(msg, val) */ 
EMSPutUint32(foo, msg, fileld); 
EMSPutUint32(foo, msg, pageNumber); 

foo->SendMsg(msg); /* msg is destroyed now! (maybe should sent &msg?) */ 
reply = foo->GetMsg(); /* in the synchronous case; this blocks w/ timeout*/ 
size = EMSGetUint32(foo, reply); 

pagedata = EMSGetByteStream(foo, reply); /* this guy allocs mem for you */ 
For reuse, the above can be wrapped in its own function with the signature: 
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GetPageFromAppServer(foo, &accessToken, appld, fileld, pageNumber, &size, 
&pagedata); 

When you send your next message you should reuse 'foo\ If the server had dropped the 
connection, it is automatically reestablished. When you're done with foo, you can Re- 
set() it, and reinitialize it for another server, or free it. 

That's generally how messages are sent, on the client or server. The server has a few ex- 
tra things it needs to do at startup time: 

xdr_ssljcp_mgr = new EMSConnectionMgrJ£DRJ3PC_SSLTCP; 
EMSRegisterStack(xdr_epc_ssltcp_mgr, 4567); 

This starts the socket watcher thread (if it doesn't already exist), and tells it to listen to 
socket 4567 and when a connection is made, it sets up a EMSConnec- 
tion_XDR_EPC_SSLTCP to pull data off the network when it arrives. Requests coming 
in are handled by the appropriate layers, which were initialized globally above, so EPC 
will dispatch the incoming message as configured by EMSLay- 

erEPC::GlobalRegisterAPI( . ). The socket watcher will free these connections when 
the idle timeout expires. The only problem with this approach is that if the EMSConnec- 
tion needs initialization info that is layer-specific (can't think of any right now that aren't 
global), there could be problems. The Connections should have default initializations 
that make sense so they do the right thing when messages come in (or maybe they need to 
have a special server mode?). 

When the client uses asynchronous messaging, it must have done a EMSLay- 
erEPC::GlobalRegisterAPI( . . . ) as well to register its callbacks. When the first asyn- 
chronous EMSConnection is created, the socket watcher is started (if not already there). 
When the connection is actually established with the server, the socket watcher is in- 
formed to watch it for the reply (along with the timeout). Otherwise, there is nothing 
else the client needs to do (other than track message IDs to match replies and timeouts). 



Testing design 
Unit testing plans 

The XDR packaging utility will be put together first, but little testing needs to be done 
directly on it, since it will be pulled right out of Sun's XDR code, with minimal adapta- 
tion. The next piece to be built will be the HTTP+TCP MTL (since this can mostly be 
ripped out of our prototype), and then a test client and server will be built to test it. The 
client will send data patterns of various sizes, and the server will just send them back. 
Then the eStream Procedure Call PCL will be built and tested with another test client and 
server at this level. Finally the SSL+TCP MTL will be constructed and tested with the 
null authentication/encryption/hashing configuration, which will still test basic handshak- 
ing. Once other security issues are handled, we'll enable full SSL functionality. 
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Beyond the basic testing, certain features need specific testing: client reply timeouts [put 
sleeps m server remote procedures and also just have servers not respond at all], server 
connection idle timeouts [put sleeps in clients], transparent connection re-establishment 
p'ust have a client reuse the connection from above], connection states [need to start mak- 
ing lots of requests while a connection is being asynchronously started], message states 
[different amounts of stuff on the queues in & out], reuse of connection objects. 

Then error conditions need to be tested, since if EMS hangs, the whole system is effec- 
tively down. Evil test clients will drop connections, even get killed, send garbage data 
with bad headers (various possible fields), send huge messages. Of course, implementers 
of EPC functions need to sanity/validity check any incoming parameters of those func- 
tions. Evil test servers will respond with bad reponses as well. Coverage tools will likely 
come in handy, since much of EMS 's code will be present to check for all kinds of eirors. 

Stress testing plans 

Stress testing is very important for EMS. To satisfy this need, I'll need a server that im- 
plements remote procedures for the various test cases above, across more than one EPC 
API, and then run a bunch of the above singular clients all at once, with multiple in- 
stances of each. That will stress EMS on the server side, and should be run for many 
minutes (along with evil test clients) to try to expose any memory leaks and other corrup- 
tion. Additionally, I'll build a client that communicates with multiple servers, basically 
taking the above test clients and running them in different threads in the same process, 
thereby stressing the single shared EMS code on the client side. > 

Cross-component testing plans 

EMS's interaction with other components that needs to be tested besides the external in- 
terface is its interaction with proxy servers and firewalls. We'll have to implement some 
kind of test bed that resembles a real world setup and run EMS through it, perhaps with 
different potential configurations. We needs somebody familiar with proxy servers and 
firewalls to look at this. 

Upgrading/Supportability/Deployment design 

EMS will report errors to the caller as they occur (and log them if they're serious). Some 
errors may be fatal to the component in which case it'll have to do the right thing (do we 
have a common "out of memory" handler or something like that we care about?). Of 
course for debugging, there will be debug logging, but I'm not sure we want to put run- 
time checks for that in or not. 

There are going to be rules and assumptions for the EMS layers that I'll have to describe 
so that future additional layers will conform and thus integrate properly. I'll add these to 
the document here once those rules are solid, which won't happen until most of the im- 
plementation is complete. 
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Open Issues 

1 . Need to figure out how to integrate turning off TCP send coalescing (or see if 
SSL already uses it). 

2. How do shutdown of EMS system? (wait for EPC worker threads, etc?) 

3. Still need to think about how newly formed connections work on the server. 
Likely only to fully understand once implementation is underway. 

4. RealPlayer can get the proxy address from the local installed browser (so it 
claims), probably from the registry. Maybe we should do this too? Does this 
apply to SSL proxies as well? 



Omnishift Technologies, Inc. 



16 



Company Confidential 



Server Installation Requirements 
Author: Bhaven Avalani 
Omnishift Confidential 

Pre-requisites: 

1. Hardware: 

• 3 dual-processor machines with the following configurations: 

i. 400 Mhz (or higher ) dual-processor CPU. 

ii. 128 MB (or higher) RAM. 

iii. 20 GB (or higher) disk space. 

iv. 1 0 GB (or higher) disk space on C: drive. 
(We shall call these machines A,B and C for further discussions). 

2. Software: 

• Each of the machines should be loaded with Widows 2000. 

• Machine A to be configured with SQL Server 7.0. 

• Machine B to be configured with Apache( 1 .3 . 1 2) webserver and 
Tomcat(3 . 1 ) servlet engine. 

• Machines A,B and C to be configured with SQL Server(7.0) clients. 

3. Software shipped by Omnishift: 

• Web Server Software: 

i. DDL script to create the EStream data model. 

ii. webserver.jar ( Servlets to access the eStream database). 

iii. Sprinta2000.jar ( JDBC driver from Inet software). 

iv. JSP and HTML pages for the Web server applications. 

• AppServer.exe 

• SlimServer.exe 

• Monitor.exe 

• DLLs: 

i. otdbdll.dll ( Dll for ODBC database connectivity). 

Installation: 

1 . The software provided by Omnishift will be in a zip format (for Alpha). Unzip 
the Omnishift software. This will create the following subdirectories: 

a. C :/eStream 1 .0/bin (All the exes and dlls will go here). 

b. C:/eStreaml.0/logs . 

c. C./eStreaml .0/conf (to store the flat file configurations). 

d. C:/eStreaml.0/ess (to store the estream sets). 

e. C:/eStream 1 .0/esc (to store the estream cache). 

f. C:/eStreaml .0/web (to store all web server related components). 

2. Create the eStream database using the data model script supplied on Machine A. 

3. Configure the eStream ODBC data source on each of the machines. 

4. Configure the Apache/Tomcat server with eStream software. 

5. Configure the physical machines using the Admin UI. 

6. Configure the monitor services on machines A, B and C. 

7. Configure the monitor setting in the database using the Admin UI. 

8. Configure Slim Servers on machine A and C. 



9. Configure App servers on machine B and C. 

10. Deploy the eStream sets on C:/eStreaml .0/ess directory. 

1 1 . Start the Slim and the App servers. 

Development setup: 

1 . Install Visual Cafe and Dreamweaver from 
Wwkstl8\download\WebGainStudio40(Trial).exe or http://www.webgain.com/. 

2. Install Tomcat 3 J from http://jakarta.apache.Org/builds/tomcat/release/v3.l/bin/. 

3. Install SQLServer 7.0. 

4. Install JDBC driver from inet software. Wwkst 1 8\download\sprinta2000 trial.zip . 

5. Install JDK 1.3 from http://java.sun.eom/i2se/l .3/ . 

6. Set environment variable TOMCATHOME and JAVAHOME to your Tomcat 
and JDK root directory (e.g. c:\tomcat and c:\jdkl .3). 

7. Add JDBC driver path (e.g. c:\JDBCDriver\Sprinta2000.jar) and eStream 
application path (e.g. c:\tomcat\webapps\classes\webserver.jar) to your 
CLASSPATH. 

8. Add eStream application into tomcat's server.xml file (e.g. add "<Context 
path=7estream" docBase-'webapps/estream" debug= M l" reloadable= M true" 
x/Context>" with other sample applications in c:\tomcat\conf\server.xml file). 

9. In Project\Option of Visual Cafe, point the deployment directory and path to your 
tomcat's eStream directory and file (e.g. c:\tomcat\webapps\classes and 
webserver.jar), and put JDBC driver path into project directories (e.g. 
c:\JDBCDriver\Sprinta2000.jar). 



Server Installation Requirements 
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Pre-requisites: 

1 . Hardware: 

• 3 dual-processor machines with the following configurations: 

i. 400 Mhz (or higher ) dual-processor CPU. 

ii. 128 MB (or higher) RAM. 

iii. 20 GB (or higher) disk space. 

iv. 10 GB (or higher) disk space on C: drive. 
(We shall call these machines A,B and C for further discussions). 

2. Software: 

• Each of the machines should be loaded with Widows 2000. 

• Machine A to be configured with SQL Server 7.0. 

• Machine B to be configured with Apache(l .3.12) webserver and 
Tomcat(3.1) servlet engine. 

• Machines A,B and C to be configured with SQL Server(7.0) clients. 

3 . Software shipped by Omnishift: 

• Web Server Software: 

i. DDL script to create the EStream data model. 

ii. webserver jar ( Servlets to access the eStream database). 

iii. Sprinta2000.jar ( JDBC driver from Inet software). 

iv. JSP and HTML pages for the Web server applications. 

• AppServer.exe 

• SlimServer.exe 

• Monitor.exe 

• DLLs: 

i. otdbdll.dll ( Dll for ODBC database connectivity). 

Installation: 

1 . The software provided by Omnishift will be in a zip format (for Alpha). Unzip 
the Omnishift software. This will create the following subdirectories: 

a. C:/eStreamL0/bin (All the exes and dlls will go here). 

b. C:/eStreamL0/logs 

c. C:/eStreaml .0/conf (to store the flat file configurations). 

d. C:/eStreaml .0/ess (to store the estream sets). 

e. C:/eStreaml .0/esc (to store the estream cache). 

f. C:/eStreaml .0/web (to store all web server related components). 

2. Create the eStream database using the data model script supplied on Machine A. 

3. Configure the eStream ODBC data source on each of the machines. 

4. Configure the Apache/Tomcat server with eStream software. 

5. Configure the physical machines using the Admin UI. 

6. Configure the monitor services on machines A, B and C. 

7. Configure the monitor setting in the database using the Admin UI. 

8. Configure Slim Servers on machine A and C. 



9. Configure App servers on machine B and C. 

10. Deploy the eStream sets on C:/eStreaml .0/ess directory. 

1 1 . Start the Slim and the App servers. 

Development setup: 

1 . Install Visual Cafe and Dreamweaver from 

Wwkstl 8\download\WebGainStudio40(Trial).exe or http://www.webgain.com/. 

2. Install Tomcat 3,1 from http://jakarta.apache.org/builds/tomcat/release/v3. 1 /bin/. 

3. Install SQLServer 7.0. 

4. Install JDBC driver from inet software. Wwkst 1 8\download\sprinta2000 trial.zip . 

5. Install JDK 1 .3 from http://iava.sun.eom/i2se/l .3/ . 

6. Set environment variable TOMCAT_HOME and JAVA HOME to your Tomcat 
and JDK root directory (e.g. c:\tomcat and c:\jdkl .3). 

7. Add JDBC driver path (e.g. c:YFDBCDriver\Sprinta2000.jar) and eStream 
application path (e.g. c:\tomcat\webapps\classes\webserver.jar) to your 
CLASSPATH. 

8. Add eStream application into tomcat's server.xml file (e.g. add "<Context 
path=7estream" docBase- 'webapps/estream" debugs" 1" reloadable= M true" 
x/Context>" with other sample applications in c:\tomcat\conf\server.xml file). 

9. In Project\Option of Visual Cafe, point the deployment directory and path to your 
tomcat's eStream directory and file (e.g. c:\tomcat\webapps\classes and 
webserver.jar), and put JDBC driver path into project directories (e.g. 
c:\JDBCDriver\Sprinta2000.jar). 
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Functionality 

The Software License Management (SLiM) server is required to enforce licensing terms and 
track overall application usage. Its primary function is to grant, renew and expire access tokens 
record application usage and aid in server load balancing. Its design can be broken down into 
three somewhat orthogonal axis: 

1 . Detailed specification of eStream client interfaces - the need. 

2. Design and usage of Server Comm on Services (CSC) & server database -the tools of the 
trade. These include logging, system monitoring, thread package, encryption, TCP/IP 
communications, etc. 

3. Core SLiM server logic that fulfils client interfaces (1) using CSC (2). 

This document address items 1 and 3 in detail; item 2 should be covered in various other 
documents emerging from the server team. 



Data type definitions 

Common Data Types 

□ Standard atomic data types everyone (clients, builders, servers) must agree on: Int8, 
Intl6, Int32, Int64, ulnt8, ulntl6, ulnt32, ulnt64, ulntl28 (specially for GUIDs). 

o Notice: No floating point types; I don't see a compelling reason to pass floating- 
point number across wire. 

□ String is represented as a size (ulnt32) and it contents. Its contents must include a NULL 
termination character, it must be included as part of the size field. 

| Lenpth | characters | 

□ All eStream sets will use little endian representation. 
All complex data structures between major components (def: at least client/servers) 
should be version identifiable. Proposal: put a version number (uint32) as 1 st word in the 
structure. 

Most complex structures are variable length because they contain strings. I think it would 
be good to put the total size (in bytes) of that structure as second word so that reader can 
know how much to read. If marshalling/unmarshalling utilities provide a way to represent 
that, it won't be needed in each struct. 

We'll need a single place where all globally (definition: across client and servers, and 
between servers) visible macros (#define & enum) are defined. We'll also need reserved 



□ 
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name spaces, and reserved number ranges for different components. Until all that is 
decided, I am defining macros without assigning any values. 
□ I am assuming that our message pack/unpack utilities will deal with alignment issues. 

Server Sets 

EstreamServerlD contains server parameters clients need to know in order to initiate a 
connection. EStreamServerSet is simply a list of individual server Ids. The main use of this data 
structure is for SLiM server to return a list of app servers that can serve a given application. 
Server ids and server sets are specific to each application; client is responsible for keeping a map 
of app id -> server set. 



#define SERVERTYPEAPPLICATION 
#define SERVERTYPESLIM 
#define SERVER_TYPE_ASP_WEB 

typedef struct { 

ulnt32 Version; 

ulnt32 SizelnBytes; 

ulnt32 MachinelP; 

ulntl6 MachinePort ; 

String MachineName; 
} eStreamServerlD; 



example of an eStreamServerlD: 

{1,20, 0x12348, 80, {12, "slO.asp.com"}} 

typedef struct { 

ulnt32 Version, 

ulnt32 SizelnBytes, 

ulnt32 ServerType; 

ulnt32 ServerlDCount, 

eStreamServerlD Server ID [] ; 
} eStreamServerSet; 

example of an eStreamServerSet: 
{!,??, SERVER_TYPE_APPLICATION, 2, 

{1,20, 0x12348,80, {12 , "alO . asp . com" } } , 

{1,20,0x12349,80, {12, "all.asp.com"}}} 

Access Tokens 

This is a main data structure that is getting passed back and forth between a client and SLiM/App 
servers. Granting an AccessToken is an acknowledgement of client's legal right to run the 
application - the license. Denying an AccessToken is an acknowledging that the client does not 
have rights to run the application; probable causes include a user running multiple sessions, user 
not paying bills etc. 
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From client's perspective, it is totally opaque; but SLiM server uses it to pass information to the 
app servers so that the app server does not have to rely on the database lookups. Each access 
token will have a unique ID. 

Terminology 

Billing Granularity - Granularity at which an ASP is interested in billing its customers. Most 
ASPs today bill on monthly basis and eStream will assume that to be the 'norm'. However, to 
support things like short trial memberships, we'll design eStream to handle billing as often'the 
AccessToken Renewal Frequency (defined below). If an end user simply purchases the 
eStreamed application, the billing granularity is infinite - the upper bound. EStream should not 
assume that billing granularity for all apps served by an ASP is the same. 

AccessToken Renewal Frequency -Frequency at which the client must renew its access tokens 
in order to continue eStream application use. This must be tunable parameter whose upper bound 
is the billing granularity; it is also the smallest billing granularity we'll support. Not all access 
tokens are required to have the same renewal frequency. 
Recommendation: 10 minutes. 
Tradeoffs: 

1 . This is the smallest granularity at which a client can be evicted (defined below). 

2. Finer granularity may increase the number of hits to the SLiM server and adversely effect 
its scalability. 

Eviction Notice - In general there will be times when an ASP wants to stop a user from using an 
eStream application, which also means stopping a user from consuming ASP server resources 
Possible reasons may be: 

• Lack of payment. 

• Termination of a trial membership. 

• To force the client into upgrading an app. 

• Just because the restroom is freezing cold. 

EStream infrastructure has an inherent limitation that servers can't push anything on the client 
That means SLiM servers must deny an access token or its renewal, to effectively deliver an 
eviction notice to the client. Also, App servers may need to be informed of such evicted access 
tokens so that they can deny paging requests. 

Decision: After looking at some scalability numbers, we concluded that a renewal frequency of 
10 minutes should not affect the overall performance and scalability of eStream system 
Consequently, we don 't have to communicate the list of evicted tokens to the app server 
since they would be invalid soon (avg 5 minutes) anyways. This simplifies server designs 
by reducing cross communication between slim servers and app servers. 

typedef struct { 

ulnt32 Version; 
ulnt32 SizelnBytes; 

ulntl2 8 ATID; // AccessToken ID - QUID 

String Userld; // GUID. 
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ulntl28 AppID; // GUID. 

ulnt64 IssueTime; // POSIX time_t format. 

ulnt64 ExpirationTime; 
} eStreamAccessToken; 

Other Common Data Structures 

In order to allow easy/automatic updates of eStream application, we need to define a protocol by 
which a client can be informed of app updates. This structure will also be used when installing 
subscribed applications on a client. 

AppName, VersionName - describe the application. 

Message - a short description of an application. 

Flages, such as ForcedUpgrade - client must upgrade the application. 

RootFileNumber - is sort of the version # of an application root directory. 

RootFileMetadata - metadata of the root directory. 



typedef struct 


{ 


ulnt32 


Version; 


ulnt32 


SizelnBytes; 


ulntl28 


AppID; 


String 


AppName ; / / 


String 


VersionName; // 


String 


Message; 


ulnt8 


ForcedUpgrade ; 


Int32 


Roo t Fi 1 eNumber ; 


???? 


RootFileMetadata ; 


} eStreamAppInfo; 



Interface definitions 

In a single process context, cross-module interfaces are easy and intuitive when defined as 
C/C++ procedure calls. However, for client/server (and perhaps server/server) interfaces we 
need to define our own RPC-like protocol. Sameer covering this (EMS - estream messaging 
services) in a different design, but I want to state couple of assumptions I am making: 

• Each EMS call is assigned a unique number (Int32). Codes must be uniform across all 
servers (i.e. no duplication of names and numbers). We should reserve some namespaces 
and numbers for each eStream server. Following is the current list of EMS codes between 
SLiM/Clients. 



#define 
#define 
#defme 
#define 
#define 
#defme 
#define 



EMCC_NULL 

EMCC_ACQUIRE_ACCESS_TOKEN 

EMCC_RENEW_ACCESS_TOKEN 

EMCC_RELEASE_ACCESS_TOKEN 

EMCC_REFRESH_SERVER_SET 

EMCC_GET_LATEST_APP_INFO 

EMCC_GET_SUBSCRIPTION LIST 
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In addition to any data (pages etc.), an EMS calls needs to return a number of codes to 
communicate success/errors. Following structure provides a container for returning 
multiple return codes. By convention, we'll put either EMCR FAILURE or 
EMCR_SUCCESS in Code[0]. 

typedef struct { 

ulnt32 SizelnBytes; 

ulnt32 RetumCodeCount; 

ulnt32 ReturnCodes[]; 
} EMCReturnCodes; 

#defme EMCRSUCCESS 

#defme EMCRFAILURE 

#define EMCR_USER_AUTH_FAILED 

#defme EMCR_ACCESS_TOKEN_INVALID 

#define EMCR_SUBSCRIPTION_INVALID 

#define EMCR_LICENSE_NOT_AVAILABLE 

#define EMCR_LICENSE_ALREADY_HELD 

#define EMCR_EVICTION_NOTICE 

#define EMCR_EVICTION_MUST_UPGRADE 

#define EMCREVICTIONENDMEMBERSHIP 

#define EMCREVICTIONNOPAYMENT 



Acquire Access Token 

Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: RPCC_ACQUIRE_ACCESS_TOKEN 

m ulntl28 SubscriptionID 

m String UserName 

m String Password 

OUT EMCReturnCodes RetumCodes 

OUT eStreamAccessToken AccessToken 

0UT ulnt32 RenewalFreq 

OUT eStreamServerSet AppServerSet 

OUT eStreamAppInfo LatestAppInfo 

Client will use this interface prior to starting an eStream application to grab the license It 
accepts a subscription id, which clients received when an app was subscribed, and password- it 
replies with at least a list of return codes and possibly, the access token, its renewal frequenc'y 
and a set of servers that can serve this app. 

AccessToken - Client treats them as opaque data structures and renews them within its renewal 
frequency. 
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RenewalFreq - This ulnt64 is the number of seconds the access token is valid for once it 
receives it. You probably don't want an absolute count (i.e. # of seconds since epoch) since 
clients can interpret it differently due to clock skew. 

ServerSet - When a client gets an access token, it will be given a list of app servers that can 
serve the particular app. The ServerType member of eStreamServerSet structure will be 
SERVERTYPEAPP. The list is specific to each app and should be managed as such. 
LatestAppInfo - SUM servers will pass information about the latest app version (root) using this 
structure. Refer to client eFS design for more detail. This structure will always be passed- client 
will ignore it if it already has the latest version. 

Note: There is a big difference between major and minor upgrades: a major upgrade 
would be going from word 98 to work2000 (where app ids must change) where as a 
minor upgrade (app ids will not change) means applying a patch or a service pack. 
LatestAppInfo tries to transparently propagate latter (minor upgrades) to end users 
without requiring end users to unsubscribe/subscribe apps. Major upgrades will require 
end users to go back to the ASP web server and change subscriptions. ASP can force the 
end user into changing subscriptions (word 98 to word 2000) using 
EMCR_EVICTION_MUST_UPGRADE error code. 

ReturnCodes 

Success: EMCRSUCCESS 

Failure: EMCRFAILURE, plus one of following: 

• EMCR_USER_AUTH_FAILED - Can't authenticate user with specified passwd 

• EMCR_LICENSE_NOT_AVAILABLE- License is not available 

• EMCR LICENSE ALREADY HELD - If the user is already holding the license, SLiM 
server returns this error code along with the access token that is held & its renewal 
frequency. Most common cause of this error is when an end user tries to run an eStream 
app on two different machines simultaneously. NOTE: returned token doesn't give the 
right to run the application and should be treated as a denial of access token. Reason for 
returning the token/renewal interval is to allow the client software can effectively release 
the token, wait some time (>= renewal frequency) and re-try. 

o The reason client has to wait is because SLiM servers will not communicate the 
list of 'bad' access tokens to the app server. 

• EMCR_EVICTION_NOTICE -ASP wants to stop the user from using ASP resources. 
Server may also add code that describe the reason like 'no payment' etc. Note that no 
access token will be given! This may change in the future to allow some grace period 

o EMCR_EVICTION_MUST_UPGRADE - This type of eviction means the ASP 
wants the end user to stop using this particular application in favor of another 
(major) version of it. For example, Word 98 to word 2000. 

Renew Access Token 

Caller: eStream Client 

Callee: SLiM Server 

RPC Code: RPCC_RENEW_ACCESS_TOKEN 

m String UserName 

*N String Password 
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IN/OUT 



eStreamAccessToken 
EMCReturnCodes 
eStreamServerSet 
ulnt32 



AccessToken 
ReturnCodes 
AppServerSet 
RenewalFreq 



OUT 
OUT 
OUT 



Clients will use this interface to renew the access token before it expires. Client will specify the 
old access token and if there are no errors, get back EMCR_SUCCESS, a new access token, new 
app server set (ServerType field of eStreamServerSet structure will be SERVERTYPEAPP) 
and new renewal frequency. Upon getting the new app server set, client must remove the old app 
server set for this application. If for some reason, the access token is expired, SliM server will 
treat this request as 'Acquire Access Token' and may return error codes possibly from that 
interface (this is one of the reason for asking for usernames/password). 

NOTE: Unlike Acquire Access Token, it is not returning LatestAppInfo or 
EMCR_EVICTION_MUST_UPGRADE error codes because once the app is running, we can't 
upgrade apps while running. 

ReturnCodes 

Success: EMCR_SUCCESS 

Failure: EMCR_FAILURE, plus one of following: 

• EMCR ACCESS TOKEN INVALID - Access token is invalid. 

• EMCR_EVICTION_NOTICE -ASP wants to stop the user from using ASP resources. 
Server may also add code that describe the reason like 'no payment' etc. 

o NOTE: will NOT return EMCRJBVICTIONJVfUSTUPGRADE. 

• Error codes from Acquire Access Token. 



Release Access Token 

Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: RPCCJRELEASEACCESSTOKEN 



Client will use this interface to release the license held by the specified user. It should be called 
synchronously when the application exits or crashes. The reason for requiring usernames and 
password is to authenticate the identity of the caller against access token owner. The reason for 
proactively releasing tokens as opposed to just letting them expire is because releasing it allows 
the user to re-acquire it (on the same or different machines) without waiting for it to expire. This 
allows the user to do acquire -> release -> acquire without any wait. 

ReturnCodes 

Success: EMCR_SUCCESS 



IN 
IN 
IN 
OUT 



eStreamAccessToken 
String 
String 

EMCReturnCodes 



AccessToken 
UserName 
Password 
ReturnCodes 
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Failure: EMCRFAILURE, plus one of following: 

• EMCR_ACCESS_TOKEN_INVALID 

• EMCRUSERAUTHF AILED 



Refresh App Server Set 



Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: RPCC_REFRESH_APP_SERVER_SET 

IN eStreamAccessToken AccessToken 

IN ulnt8 BadQoS 

IN ulnt8 NoService 

OUT EMCReturnCodes ReturnCodes 

OUT eStreamServerSet ServerSet 



App Server sets are given to a client when an access token is acquired and are automatically 
refreshed when an access token is renewed. However, the client can always refresh its app 
server sets using this interface. Potential reasons for clients to do this: 

- All servers in the current server set are not responsive - NoService = TRUE 

- Servers are up, but client experiences bad QoS (network delays/timouts). BadQoS = 
TRUE 

ReturnCodes 

Success: EMCR_SUCCESS 
Failure: EMCR FAILURE, plus one of following: 

• EMCRACCESSTOKENINVALID 



Get Subscription List 

Caller: eStream Client. 

Callee: SLiM Server. 

RPC Code: EMCC_GET_SUBSCRIPTIONJLIST 

IN String UserName 

IN String Password 

OUT EMCReturnCodes ReturnCodes 

OUT ulnt32 NumberOfSubscriptions 

OUT ulntl28[] SubscriptionID[] 

A client can ask for the current list of subscribed applications using this interface. SLiM server 
returns the number of apps subscribed and an array of subscription ids. 

ReturnCodes 

Success: EMCR_SUCCESS 
Failure: EMCR FAILURE, plus one of following: 
• EMCR_USER_AUTH_F AILED 
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Get Latest Application Info 



Caller: 

Callee: 

RPC Code: 

IN 

IN 

IN 

OUT 
OUT 



eStream Client. 
SLiM Server. 

RPCC_LATEST APP INFO 



String 
String 
ulntl28 

EMCReturnCodes 
eStreamAppInfo 



UserName 

Password 

SubscriptionID 

ReturnCodes 

Upgradelnfo 



Any upgrades pending? This functionality is piggy backed on 'acquire access token interface, 
but there is some value in providing it as an explicit interface. SliM server will give you the 
latest application information block associated with the specified subscription id; the client can 
decide if it already has the latest root (version) or not. 

ReturnCodes 

Success: EMCR_SUCCESS 

Failure: EMCRFAILURE, plus one of following: 

• EMCR_USER_AUTH_F AILED 

• EMCR_SUBSCRIPTION INVALID 



Component design 

Server Common Services 

Following diagram shows the common portion of all eStream servers. Most of these boxes won't 
be described in this document because they are covered in specific documents. 



Logging 




Caching 


Command 
Line 




Config 
Manager 






MainLoop 




Exceptions 






I18N 




Encryption 






Thread 
Mgmt 






Compression 


Core 



Load 



Heartbeat 



Server State 



Management 



SERVER COMMON SERVICES 



TCP/IP 



HTTP 



Communication 




Various Decisions 

• SLiM server will uses an ODBC interface to communicate with the central database. 
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• From eStream client's perspective, all SLiM servers are equal in functionality. This is 
unlike an application server, which can be segmented to serve specific applications. 

• Each SLiM server will have a unique IP/Port combination. Multiple SLiM servers 
running on the same machine can be distinguished by giving them different port numbers. 

• SLiM servers (and app servers) will not assume any default ports; it will rely on an ASP 
admin to configure the port assignment. With help from OTI, ASP admin will determine 
how many eStream servers need to run on a machine and assign a unique number to each 
eStream server. 

Hardware Failover 

There will be a pool of SLiM servers at an ASP site; from eStream client's prospective, each of 
them is identical. When a client subscribes to an eStream application, it gets a set of SLiM 
servers to communicate with. The clients will keep this list in memory and refer to it when 
calling SLiM server interfaces. If it experiences difficulty communicating with a particular SliM 
server, it will try other servers that are part of the server set. If for some reason the server set is 
lost, or all servers in the set are not responding, a client can always go back to the ASP web 
server and refresh its server set. This gives you a transparent (from end user's prospective) 
hardware fail over path. 

The same approach will also work for app server fail-over scenarios; specific differences are 
that: 

1 . SLiM Servers, not ASP web servers, will provide the app server set. 

2. App server list will be refreshed automatically, when access tokens are renewed. This 
allows ASP admins to take out servers from the pool by waiting certain amount of time 
(>= access token renewal frequency) and not cause unnecessary client timeouts. 

3. App server sets are specific to each app; SLiM servers are not. 



Load Balancing 

In eStream 1.0, we will not require a third party load balancers at an ASP site; we'll do minimal 
things at both ends (clients/server) that should be good enough for small to medium size ASPs. 
We may have to test with selective 3 rd party load balancers to see if we can work with them or 
not; but this is an open issue (listed in the open issues at the end of doc). 

In eStream 1.0, we'll capitalize on the hardware fail-over mechanism to also aid load balancing. 
Following two actions will perform load balancing: 

• When a client gets server sets (app or SLiM servers), it will distribute its hits randomly 
among the server in the set. In addition, clients will also get new app server sets every 
time they renew access tokens. 

• On server side, the monitor will keep track of each server's response times to process 
client's requests. The data gets soiled from most responsive to least responsive and stored 
into the database. Top 'X' servers from this list will be given to the client when it makes 
an explicit request to refresh its app server set, acquires or renews an access token. 
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Testing design 



Interface Testing 

SUM server is tightly coupled with three components: client, ODBC/Database and monitor; it is 
fairly difficult to isolate it from all of those components for unit testing. A better approach is to 
exhaustively test the client/SLiM server interfaces, which will in fact also test large numbers of 
interfaces to the other two components. The idea is to crank up a client that will make every 
possible SLiM server request and make sure that SLiM server responds accordingly. 

I think it is good idea to create a simple testing framework (that may evolve with time) that will 
simulate a real client to SLiM and app servers. We can do this by writing a program that includes 
common (client/server) data structures definitions, links in our eStream Message Services (EMS) 
component and invokes various interfaces like 'Acquire Access Token'. From SLiM server's 
prospective, this test program is a working client. 

For each client/server interface (i.e. Acquire access token) write a test case (dummy client) that 
will: ' 

o Assume that we have created a dummy database that has certain users, passwords and 
subscriptions. 

o Invoke SLiM server with all possible input permutations. This isn't too bad since 

most interfaces have 2 to 4 arguments, 
o In the process, ensure that SLiM server returns all possible return values it can. 

For instance, lets assume that Acquire Access Token has following prototype- 
AET(uIntl28 subID, String UserName, String Password); 

TEST BEGIN: 

Assert (AET(NULL, NULL, NULL) returns 

EMCRFAILURE & EMCRUSERAUTHFAILED); 

Assert (AET(good_sub_id, good user name, good_password) returns 
EMCR_SUCCESS, an access token, its renewal freq. Etc.); 

Assert (AET(good_sub_id, good_user_name, good_password) returns 
EMCR FAILURE & EMCR LICENSE ALREADY HELD); 



Stress testing plans 

Stress testing in general will be common across all eStream servers. I think it will be a good idea 
to invest in a 3 r party tool that can simulate real-time load on eStream servers and see its 
responses. Rational has various tools such as Visual Test, Robot and Site Load that are worth 

evaluating. 
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Coverage testing plans 

Lot of these items will apply to all eStream components and probably should be covered in a 
separate eStream test plan document. I am not sure if we should do these things before each 
component is done or wait until they are integrated. I just want to state what may be obvious so 
that it is documented: 

• SUM server will achieve 85% PFA coverage as measured by Rational PureCov. Tests 
used to measure PFA coverage will be reproducible, either by hand or via an automated 
test suite. 

• SLiM server will resolve all memory corruption and memory leak issues as reported bv 
Rational Purify. 

• We should have test cases that will exercise all command line options for SLiM server. 

• SLiM server will be code reviewed by at least two peers. 

Upgrading/Supportability/Deployment design 

This document must have a discussion of how the component addresses any specific issues 
related to upgrading, supporting and deployment of e-stream applications. Some examples 
include: error conditions detected and reported by this component, any special hooks this 
component will provide for monitoring, hints for troubleshooting problems, any special hooks 
for debugging this component. 



Open Issues 

This is a list of issues that need to be further investigated or revisited during implementation 

1. How do you produce GUIDs on unix servers? Should app ids, user ids, access token ids 
be guids or we should create them by knowing what numbers are already used? 

2. Resolve big-endian - little-endian issues. Owner: Sameer 

Meaning of 'eviction' notice is not conveyed to the end user yet. Owner: Client person - 
Ann. " r 



3. 



4. Encryption impact on SLiM servers. Owners: Amit & Igor. 



5 



Global name space & number ranges for different components. Owner: Bhaven 

6. ASCII v/s Unicode strings? Owner: Sameer. 

7. Test with 3 rd party load balancers to see if We work or not. Requirements for deployment 
team: tell us which load balancer to certify against and set them up in our future testing 
lab. Owner: deployment team. 
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eStream Web Server Load Monitoring Applet Low Level 

Design 



Jae Jung 




Functionality 

One of the requirements for the eStream web server is a facility for monitoring 
server load (eStream requirements 3,0, 3.2, 3.4, 3.5). Per this document, this 
facility will be provided by a graphical load-monitoring applet that will be 
available for deployment at customer sites as part of the eStream web server 
installation. 

The load-monitoring applet will present information in the following formats 
through a graphical interface: 

• Real-time server load information plotted on strip chart 

• Historical server load information plotted on line chart 

• Multi-server real-time or historical load information on a line chart 

Requirements 

The following list details the provisional requirements for the load-monitoring 
applet. The remainder of this design document is based on these requirements. 

1 . The applet will be able to display server loads in "real-time" as load data is 
retrieved from the server. 

2. The applet will be able to display (in a separate mode from real-time 
monitoring) historical load information to the extent that this information is 
available in the database. 

3 . The applet will be configurable (via applet parameters) for the following 
settings: 

Data retrieval rate, i.e., the frequency with which the applet request new 
load data from the web application server 

-Chart window size, i.e., the number of datapoints shown in the chart 
window at any one time. 

4. The applet will be capable of concurrently displaying the load of multiple 
servers in a clear and concise fashion. 



5. 



The applet will support the displaying of cumulative and average load 
statistics for multiple servers. 



6. In real-time mode, the applet will operate as a strip chart, with the fastest chart 
speed determined by a global configuration setting in database, typically 
corresponding to the frequency with which the eStream Monitor inserts load 
records into the database. 

7. The applet will retrieve load information from the database via an http 
connection to the eStream web app server. 

8. The applet will ran in browsers with Java 1 .0.x and 1 . 1 .x support. 

9. Internationalization support. Both the Applet and the backend pieces should 
be internationalizable. 



Description 

The load monitoring applet is comprised of two components. The first 
component will manage the retrieval of real-time or historical server load 
information from the eStream database. The second will consume this data and 
present this information to the user in a clear and concise graphical format. In 
addition to the applet, server-side objects will need to be written or extended to 
service data requests from the applet. 

For the alpha-release of the eStream web server, the graphical presentation 
component will not be written internally; rather a commercially available applet 
or package will be used. Other aspects of the applet and the server-side 
components will be implemented to facilitate transitioning to an internally 
developed graphical component if such a decision is made for later releases. 

User Interface Design 

The following two screen shots are representative of the way that the load 
monitoring applet might be used in a particular monitoring/administration Brower 
interface. The first shot shows a general server administration and monitoring UI 
and the second shows a detailed load monitoring UI within which the load 
monitoring applet will be embedded. 

The UI options shown in the second shot illustrate some of the reporting options 
for which the load monitor will be initially configurable. The applet(s) will be 
readily extensible to generate additional reports and more complex data 
combinations if desirable. 



Interface Definitions 

The load monitoring applet has two interfaces with other webserver components: 
<APPLET> tag parameters for its configuration within the web browser page and 
the HTTP request/response format with the web application server to request and 
receive load data. 



L Applet <-> HTML Page Interface (<APPLET> Tag Parameters): 



Parameter 


Req'd? 


Significance 


ripfniilt 
l^Cl alllt 


hostName 


Yes 


Host name of webserver 


None 


hostPort 


No 


Host port of webserver 


"80" (int) 


chartSpeed 


No 


Time (in seconds) between chart scroll; also 

me icouiuiiuii ui ine X - aXlS 


"5" (int) 


updatePeriod 


No 


Interval (in seconds) between HTTP requests 
for server load data. Note if chartSpeed < 
updatePeriod, the applet buffers load data for 
presentation. Maximum information latency 
will at most be equal to this value plus round 

tlljp lllilC JLvJl L11C Uaul I Ci|UCoL 


"10" (int) 


chartWidth 


No 


AVirltVl (in t\c\cst\ f\f* tViP 1 Qrvn1p>t oViarf* in 
* * x vim ^ni H\yFk.oj Ui UIC appiCL dlail, 111 

conjunction with chartSpeed, implicitly 
determines the amount of data displayed 


ju (int) 


mode 


No 


"current" for real-time monitoring and 
"historv" for historical 


"current" 


startDate 


No 


if mode- 'history", the starting date for 
which the historical load chart will be 
plotted. Note if mode="current" this 
parameter is ignored. 


none; value 
must be in Java 
Date format 


stopDate 


No 


if mode="history" the stop date for which 
the historical load chart will be plotted. Note 
if mode="current" this parameter is ignored. 
Note that stopDate override the chartSpeed 
setting insofar as x-axis resolution is 
concerned. 


none; value 
must be in Java 
Date format 


numServers 


Yes 


number of servers to be plotted 


1 (int, max 50) 


serverNamel 
serverName2 

serverNameSO 


Yes 


the id of the server(s) being plotted (up to 50 
servers can be plotted at once). This must 
correspond to a existing serverED in the 
Server database table 


none 


serverDatal 
serverData2 

serverData50 


No 


an initial set of data with which to display 
the initial chart. Note that these parameter(s) 
are the default means by which historical 
load data is displayed. Each set should be a 
comma delimited list of numbers (float) 


none 



• Note that applet tag parameters are all strings; where the applet does an 
internal type conversion, the target type format of the particular parameter 
has been noted. 

• Note that the parameters determining the appearance of the chart (e.g., line 
colors, grid painting options, custom labels, etc.) are not included in this 
list. These parameters will either be determined by the parameters 
available for configuration in the commercial charting applet or TBD at 
some later date if the charting component is developed internally. 

2. Applet <-> Web App Server Interface (HTTP) 

Input: 

The load monitoring applet will request load data from the webserver by 
executing the web server's MonitorServlet with the following parameters: 

href= ft /MonitorServlet?action=getLoadData 
&serverld=... 
&startDate=... 
&stopDate=... 
&numPoints=... 

where: 

• serverld is a comma-delimited list of one or more server(s) for which load 
information is being requested. Note that this serverld corresponds to the 
serverlD attribute in the Server database table. 

• startDate is the (exclusive) starting date (in Java Date format) for which 
the load information is being requested 

• stopDate is the (inclusive) end date (in Java Date format) for which the 
load information is being requested 

• numPoints is the number of data points requested between the start and 
stop dates. 

The applet will process the return data points as equally time-spaced points 
between the requested start and stop dates. 

In addition, if startDate=stopDate, only one load data point (i.e., the most recent) 
will be returned by the servlet. 

Output: 

The load monitoring applet will expect load information from the web server via 
HTTP response in the following XML-like format: 



<loadData> 



<serverid=a > 

<LOADLIST> 

<LOAD value=xl/> 
<LOAD value=x2/> 
<LOAD value=x3/> 

<LOAD value=xN/> 
</LOADLIST> 
</server> 
<server id=b> 
<LOADLIST> 

<LOAD value=yl/> 
<LOAD value=y2/> 
<LOAD value=y3/> 

<LOAD value=yN/> 
</LOADLIST> 

</server> 
</loadData> 

In the above example, xl to xN represents load values for the server a (by 
serverlD), and yl to yN represents load values for server b. 



Testing Design 

The load monitoring applet and related server-side Java code will need to be tested 
according to the plan outlined for other web server components in the Web 
Server/Database Low Level Design (WebServerDB-LLD.doc). Additionally, it should be 
noted that certain applet-related parameters (i.e., updatePeriod) that affect the frequency 
with which the applet requests load data from the web server are good candidates for 
tuning in order that a good balance between Ul/measurement response and web server 
response performance be struck. 

Upgrading/Supportability/Deployment Design 

Upgrading/Supportability/Deployment Design will follow the model described for the 
Web Server in the Web Server/Database Low Level; Design (WebServerDB-LLD.doc). 

Open Issues 

1 . How much will it cost to deploy the chosen commercial java applet/package(s) as an 
OEM installation? We need to find this out before we decide on a commercial 



package. Also, another criteria for the choice would be: will we get support. Do we 
get the source code too? 

Longer term, where's the cost/benefit breakpoint for the above where it makes more 
sense to write our own charting applet/package? 
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eStream Web Server/Database Low Level Design 
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Functionality 

The eStream solution provides a set of account, user, and subscription manage- 
ment utilities. These utilities are provided as extensions to the ASP's(Application 
Service Provider) web server. 

There are three categories of users for these utilities: End User, Group Adminis- 
trator and ASP Administrator. The roles and the capabilities of each of these us- 
ers are detailed below. 

End user for a system is the user who will actually access eStream application us- 
ing the eStream clients. An end user should be able to: 

• Create Account and User attributes. (Username, Password, etc.) 

• Change Account and User attributes. * ><v 

• View all available applications in the eStream system. 

• Subscribe/Manage eStream applications. 

• View Account Status. 

1 . List of applications subscribed. 

2. Status of current subscription. 

3 . View/Change Billing information. 

4. View/Change Account information. 

A Group Administrator is an administrator for a group of users. An individual 
user is by definition a group administrator for a single user group. Capabilities of 
a group administrator are: 

• (All of single user capabilities). 

• Add delete users from a group. 

• Manage the active sessions for a group. A group manager should be able 
to release licenses from active sessions, thereby kicking out active users. 

• View the billing information. This will probably need hooks to an external 
billing system. 

An ASP administrator manages the overall application system. Capabilities of an 
ASP administrator are: 

• Manage accounts/users/subscription for all users/groups in the system. 

• Manage the application data for a subscription system. 
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1 . Add new applications to the system. 

2. Modify application information for the system. 

3. Provide the pricing mechanism for the applications^). 
• Manage the servers in the system. 

1 . Configure a server. 

2. Stop/Start a server. This is accomplished by a message to the 
Monitor server. 

3. Get load information for a server. 

4. Get logging information for a server. 

There are essentially two different types of accounts, which the system will support- Sin- 
gle user account and corporate accounts. 

The following licensing mechanisms will be supported by the system. 

• Fixed Duration License. (Typically monthly license). 

• Indefinite License. 

Description 

There are several key issues that need to be determined for the Web Server architecture 
The options available in the market to implement these technologies are listed below. 

Web Server: 



• Apache 

• Netscape Server 

• Microsoft Internet Information Server 
CGI Technology 

• Servlet/JSP 

o Tomcat ( from Apache group) 
o JRun (from Allaire) 

• Active Server Pages (available on NT only) 

• NSAPI ( C level API available for Netscape and Apache). 

• ISAPI ( C level API available for IIS and Apache) 

• CGI(Perl/Cetc). 

Database Connectivity 



• JDBC. 

• ODBC 

• Native. 
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Database 



• SQLServer 

• Oracle 

• Sybase 

• Informix 

• LDAP(??) 



The overall proposed solution for eStream 1 .0 Webserver release is: 

Apache + Tomcat(for JSP/ServIet) + JDBC + SQLServer. 

The reasons for choosing this combination for the servers are as follows: 

1 . JSP/Servlet is the only technology which is available for cross-platform and cross 
Webserver support. 

2. We need to decide on a single web server to develop and test against for release 

1 .0. Apache is chosen to be the one as it is popular on Unix and NT platforms and 
it is freely available. 

3. Tomcat(Apache group's reference implementation for JSP/Servlet specs) is the 
preferred CGI technology as it works well with Apache and all other web servers. 

4. JDBC is preferred for database connectivity as its database neutral and works well 
with Java environment of Servlets. 

5. SQLServer is the preferred database for release 1.0. This contains the scope for 
testing and deployment for eStream 1 .0. 

Since all other servers(App Server, SLM Server and Monitor) are C++ components, the 
following technology combination will be available for Database Access. 

ODBC + SQLServer. 



The data model for the eStream 1 .0 database essentially consists of two high level com- 
ponents. The database deployment architecture is shown below: 
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Server Management Component: This component's primary responsibility is to man- 
age the configuration, load and log information for a logical server in the system The 
clients to this component are all the servers and administration manager. A detailed list of 
interfaces for this component is described in the interfaces section. 

User/Account/Subscription Management: This component is responsible for maintain- 
ing the user account and subscription information for the system. The end user using the 
end user interface performs the updates to this component. Slim Server will access this 
component to validate subscriptions. A detailed list of interfaces for this component is 
described in the interfaces section. 

Group Management: This component is useful for managing groups of users. The group 
administrator can only perform updates to this component. A detailed list of interfaces for 
this component is described in the interfaces section. 

Billing Management: This component's responsibility is to provide interfaces to an ex- 
ternal billing system. A detailed list of interfaces for this component is described in the 
interfaces section. 
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Application Management: This component's responsibility is to provide application 
management interface. This component is accessible for updates only by the ASP admin- 
istrator. A detailed list of interfaces for this component is described in the interfaces sec- 
tion. 

License Manager: This component's responsibility is to manage the licenses. SLiM 
server will check out licenses from the license manager. A detailed list of interfaces for 
this component is described in the interfaces section. 



The architecture for the Web Server extensions implementation is shown below: 



Reques^ 



Dispatch Serviet 



Web 
Server 



Response 



J* V 



JSP 



JSP 



JSP 



Worker Bean 



Worker Bean 



Data Access 
Bean 



The basic elements of this architecture are as follows: 

1 . Every request into the system goes through a dispatcher serviet. This serviet will 
perform initialization, initial validation of the request and miscellaneous checks 
.before dispatching the request to a JSP page. A worker bean will responsible for 
performing the initialization. The processing of the incoming request is performed 
at this stage. The request is then dispatched to an appropriate JSP page. 

2. The JSP page will invoke worker beans to access the dynamic data from the data- 
base via the Data Access Bean and the resultant page is sent back to the user. 
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This architecture is illustrated with the following example. 

1 . User sends in a request to update the usemame and password information in the 
database. Inputs are username, old password, new password. 

2. The dispatch bean will call the user(worker) bean to: 

a. Validate the user's old password. 

i. The user worker bean will make a request to the data access bean 
to access the password for the user. 

ii. The two passwords are compared and the result is returned. 

b. If the password was valid then, update the new password. 

i. Call the data access bean to update the password in the database. 

c. Else return failure. 

3. Based on the success or failure the dispatcher will dispatch the page request to the 
appropriate JSP page. (eg. error.jsp on failure and user.jsp on success). 

4. The page will invoke the appropriate the worker bean (error bean or user bean) to 
obtain the dynamic data and send the response back to the user. 

The salient features of this architecture are: 

1 . Presentation and processing logic is separate. Thus, the customer(ASP) can cus- 
tomize the look and the feel of the pages without impacting the processing logic 
as it is segregated. 

2. The data access bean is separated from the worker beans, which are primarily re- 
sponsible for the business logic. This allows us to change the data access layer (eg 
enabling LDAP access) in the future without impacting the system drastically. 



Data type definitions 



The central data structure for Web Server is the database model. The overall database 
model for user and subscription management is shown below. 
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tlllllllfli 


llllll 




mmmrn 










PK 


usaaeld 


COUNTER 


FK2 


userld 


LONG 




starttime 


DATETIME 




duration 


LONG 


FK1 


subscriptionld 


LONG 







PK 


svtecripttonjd. 


COUNTER 


FK3 


groupld 


LONG 


FK2 


use lid 


LONG 




createdaie 


DATETIME 




modffydate 


DATETIME 


FK1 


appRcationld 


LONG 




max users 


LONG 




available 


LONG 




slate 


SHORT 









PK 




COUNTER 




acoountName 


TEXT(80) 




address 1 


TEXT(80) 




address2 


TEXT(80) 




city 


TEXT(80) 




accbuntState 


TEXT(40) 




zipcode 


LONG 




phonel 


CHAR(40) 




extl 


CHAR(10) 




phone2 


CHAR(40) 




ex 12 


CHAR(10) 




fax 


CHAR(40) 




firstName 


TEXT(80) 




lastName 


TEXT(80) 




rniddfelnttial 


CHAR(10) 











PK 


licenseUsaueld 


COUNTER 


FK2 
FK1 


subscriptionld 
userld 

starttime 

endtime 

evfctionFlag 

evfctionReason 

renewTime 

renewDuratlon 


LONG 

LONG 

DATETIME 

DATETIME 

SHORT 

SHORT 

DATETIME 

LONG 




FK1 
11 



FK2 



1 



groupld 

usemame 

password 

usertype 

notes 

accountid 



LONG 

TEXT(80) 

TEXT(80) 

SHORT 

LONGTEXT 

LONG 

SHORT 



I 









PK 


groupld 


COUNTER 


FK1 


userld 
groupname 

notes 


LONG 

TEXT(80) 

LONGTEXT 



The important features of this data model are: 

Account: Table holding all the billing and contact information for a user or a group. 

User: An end user in the system. A user can optionally belong to a group. 

UserGroup: A group of users. One of the users in the group is designated as the group 
administrator. Each group has a unique account associated with it. 

Application: This table contains the data about various applications in the supported by 
the ASP. 



Subscription: This table contains entries for subscription items. A subscription item con- 
sists of user/group, application and license. 
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Usage: This table contains the runtime information for a system. SLiM server updates 
this table with access token usage data. A billing system may interface with this table to 
generate billing data. A reporting system may interface with this table to report on usage 
patterns. 

LicenseUsage: This table is responsible for recording checked out licenses in the system. 

The data model for storing the server related information is shown below: 

PK: Primary Key for the table. 

FK: Foreign Key. Used for relations between tables. 

11,12.. : Index Columns. 

Server: This table contains entries for each logical server in the system. 

Machine: This table contains entries for each physical server in the system 

Configuration: This table contains configuration entries for a given server. The configu- 
ration entries can be hierarchical in nature. Each configuration has the following format: 

Name Value 1 [Value2] [Value3] [ParentConfigld] 

Load: This table maintains the historical and real-time load information for a given logi- 
cal server in the system. 

Service: The service table is used to record to map all the app servers and the corre- 
sponding applications that they serve. 

Log: This table maintains the logs for a logical server in the system. The log messages 
saved here are "major" events in the logical server system. A detailed logs stored in a flat 
file on the physical machine containing the logical servers 



Global Data Structures: 



class DLLEXPORT Configuration 

{ 

public: 

CnfigurationO {mConfigld - -1; mServerld = -1; mParentConfigld = -1;}; 
vrtual -Configuration) {}; 
int mConfigld; 
string mName; 
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string m Value 1; 
string mValue2; 
string mValue3; 
int mServerld; 
int mParentConfigld; 

}; 

class DLLEXPORT GlobalConfiguration 

{ 

public: 

GlobalConfigurationO { } ; 
virtual -GlobalConfigurationO {}; 
string mName; 
string mValuel; 
string mValue2; 
string mValue3; 

}; 

class DLLEXPORT DBLog 
{ 

private: 

long getTimeStampO; 

public: 

DBLogO; 
virtual -DBLogO; 
string mMessage; 
int mType; 
int mPriority; 
int mServerld; 

tm* mpTime; // Time stamp in ms since epoch. 

}; 

class DLLEXPORT ServerLoad 
{ 

public: 

ServerLoadO {mpTime=NULL; mLoadType=0; mServerLoad=0; mServerId=0;} 

virtual -ServerLoadO { } ; 

tm* mpTime; 

int mLoadType; 

int mServerLoad; 

int mServerld; 

}; 



The load data structure contains the following pieces of data: 
• Server Id. Identifier for the server. 
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• Load: Response time in milliseconds. 

For the Access Token and related data structures, please refer to the SLiM server Low 
Level Design Document. The interfaces below will discuss some of the API's based on 
the these data structures. 



Interface definitions 

The interfaces exposed by various sub-components are detailed below. 

Server Management Component: 

CreateServer 

int CreateServer (ServerConflg* config) 
Input i 

Server Configuration. (data structure defined 
in the server configuration document.) 
Output: 

Unique ID for the server. 
Comments: 

Create a logical server with predefined con- 
figuration. 
Errors : 

BAD CONFIGURATION 

NVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

UpdateServerConfig 

Bool UpdateServerConfig(int serverld, String name, String value) 

Input : 

Server Id 

Config name and value 
Output: 

Unique ID for the server. 
Comments : 

Create a logical server with predefined con- 
figuration. 
Errors : 

BAD CONFIGURATION 
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INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

AddMachine 

Bool AddMachine(String name, String domain, String ip) 
Input: 

Machine name, domain and ip. 
Output: 

Success/Failure 
Comments: 

Create a physical machine entry. 
Errors : 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

SetServerLog 

Bool SetServerLog(int serverld, LogTuple log) 

Xnput: 

Server Id 

Log tuple (data structure in the Logging 
document.) 
Output: 

Success/Failure 
Comments: 

Add the log data for a server 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerLog 

LogTuple[] GetServerLog(int serverld, int maxrows = 25) 

Inpu t : 

Server Id 

Maxrows: Maximum number of rows to be re- 
turned. 

Output: 
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Array of Log tuples (data structure in the 
Logging document . ) 
Comments: 

Get the log data for a server 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServers 

ServerTuplef] GetServersO 
Input : 
Output: 

Array of Server tuples (data structure de- 
fined above) 
Comments: 

Get all the server information 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

SetServerState 

Bool SetServerState (int serverld, short state) 
Input: 

Serverld: Unique id for a server 
State: State information for a 
server. (Defined in the server framework docu- 
ment . ) 
Output: 

Bool True/False for success/failure. 
Comments: 

Update the database with current state in- 
formation for a specified server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 
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GetServerState: Obtain the last known state for a specified server 
short GetServerState (int serverld) 
Inpu t : 

Serverld: Unique id for a server 
Output: 

State: State informat ion for a server. 
Comments: 

Obtain the last known state for a specified 
server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerConfig: Obtain configuration information for a specified server 
ServerConfig* GetServerConfig (int serverld) 
Inpu t : 

Serverld: Unique id for a server 
Output: 

ServerConfig*: State information for a 
server. (ServerConfig data structure is defined 
in the server configuration document) . 
Comments: 

Obtain the last known state for a specified 
server 
Errors : 

INVALID SERVER ID 

DB ROW LOCKED 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



SetLoadData: 

void SetLoadData (int serverld, int Load) 
Jnput : 

Serverld: Unique id for a server 
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Load: Load for the server 
Output: 

Cowmen ts : 

Monitor may call this interface to persis- 
tently store historical load data. It is still 
not clear if SLM and application servers will 
store this directly themselves. 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetServerLoad: 

void GetServerLoad (int serverld, , int maxrows = 25, int** Load) 
Tnput: 

Serverld: Unique id for a server 
maxrows: Maximum number of rows to be re- 
turned. Default is 25. 
Ou tpu t : 

Load: Load for the server 
Comments : 

Obtain server component load information to 
manage load balance. 
Errors : 

INVALID SERVER. ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



FIushLoadData: 

void FIushLoadData (<tuples> LoadData) 
Input : 

LoadData tuples containing <server id, 
server load> values. 
Output: 

Comments: 
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Used to flush aggregated load data to the 
databa 
Errors : 

INVALID SERVER ID 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

User/Account/Subscription Management Component 

CreateUser. This API is used to create user record in the system. Arguments will be 
Username, Password. 

Bool CreateUser(String username, String password) 

ValidateUser. This API is used to validate user record in the system. Arguments will be 
Username, Password. 

Bool ValidateUser(String username, String password) 

CreateAccount This API is used to create account records in the system. Arguments 
will be billing address, credit card information etc. 

Bool CreateAccount(int userid, <Account Information>couple[]) 
Input: 

Username associated with the account. 

An array of names and values for the account. 
AddSubscription. This API is used by the end users/group administrators to subscribe to 
applications. 

Bool AddSubscription(int userid, Subscription Information>couple[]) 
Input: An array of names and values for the subscription. 



UpdatePassword. Used to change user information. Password, username etc. 

Bool UpdatePassword(int userid, String old-password, String new-password); 
UpdateAccount Used to update the account information. Billing Address etc. 

Bool UpdateAccount(Couple[]) 

Input: An array of name, value pairs for the fields to be updated. 
UpdateSubscription. Used to add additional time to a subscription. 
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Bool UpdateAccount(Couple[]) 

Input: An array of name, value pairs for the fields to be updated. 

GetUserRecord. Used to get current user configuration. 

Couple[] GetUserRecord (int userid) 
GetAccountRecord. Used to get current account configuration for a user. 

Couple[] GetAccountRecord(int userid) 

GetSubscriptionRecords. Used to get to subscription records in a database. End user 
may just want to verify what they are subscribed to. 

Couple[][] GetSubscriptionRecords(int userid) 

Output: An array of array of couples containing the subscription informa- 
tion for a given user. 

DeleteUser. Used to delete users who are no longer valid in the system. Typically called 
by the ASP admin. 

Bool DeleteUser(int userid) 
DeleteAccount. Used to delete un-used accounts. 

Bool DeleteAccount(int accountld) 
DeleteSubscription. Used by the ASP admin to remove subscriptions. 

Bool DeleteSubscription(int subscriptionld) 

Group Management Component 

CreateGroup. This API is responsible for creating group accounts in the database. 
Called by the group admin user. 

Bool CreateGroup(String groupName, String admin, String notes) 
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AddUserToGroup. Adds a user to a group. 

Bool AddUserToGroup(int groupid, int userid) 
DeleteUserFromGroup. Removes a user from a group. 

Bool DeleteUserFromGroup(int groupid, int userid) 
GetActiveSessions. Gets the active sessions for a group. 

Couple[][] GetActiveSessions(int groupid) 

Output: An array of array of couples containing the following information 
for each active session in the system 
Username 
Licenseld 
StartTime 
EndTime 
Subscription 

DeleteGroup(int groupid); //Deletes the group 

Couple[][JGetGroupUsers(int groupid); // Gets all the users for a given group. 



Licensing Component 



CheckoutLicense: Checks out a license. 

int CheckOutLicense(int subscription^, long* pStartTime, long* pStopTime) 

Inputs: 

Subscriptionld: Subscription id of the user. 
Outputs: 

>0 for a successful license. The output is the license usage id. 
StartTime for the license. 
StopTime for the license. 

Comments: 

The system should validate the availability of the license. 

Errors: 

INVALID SUBSCRIPTION 
LICENSE NOT AVAILABLE 
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NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 



RefreshLicense: Refreshes a license. 



int RefreshLicense(int LicenseUsageld, long* pStartTime, long* pStopTime) 
Inputs: 

LicenseUsageld: License usage id. 
Outputs: 

>0 for a successful license. The output is the license usage id. 
StartTime for the license. 
StopTime for the license. 
Comments: 

The system should validate the availability of the license. 

Errors: 

INVALID SUBSCRIPTION 
LICENSE NOT AVAILABLE 
NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 
EVICTION 



CheckinLicense: Check in a license 

Bool CheckInLicense(String username, int subscription^, int licenseUsageld) 
Inputs: 

Username: user trying to check out the license 

Subscription^: Subscription id of the user. 

LicenseUsageld: Usage id for the checked out license. 
Outputs: 

Success/Failure 
Comments: 
Errors: 

NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 



ValidateLicense: Validate that the user has a license checked out. 

Bool ValidateLicense(String username, int subscription^, int licen- 
seUsageld ) 

Inputs: 
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Username: user trying to check out the license 

Subscription^: Subscription id of the user. 

LicenseUsageld: Usage id for the checked out license. 
Outputs: 

Yes/No. 
Comments: 
Errors: 

INVALID USER 

INVALID SUBSCRIPTION 

INVALID LICENSE 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



DBAcquireAccessToken 

RPCReturnCodes DBAcquireAccessTokenQong Subscription^, long* pAccessTokenld, 
string UserName, string Password, long* pStartTime, long* pStopTime, long* Applica- ' 
tionld) 

Id of the subscription being used. 
-1 if this is a first time access. 
Username string. 
Encrypted Password 
Start time for Access Token validity. 
Stop time for Access Token validity. 
Id of the application. -1 Default. 
RPC Return codes. 

Processing: 

This is fairly complex function. The processing involved in this function call is: 

• If this is the first access (ie *pAccessTokenId = -1) then ValidateUser 

• If the Application^ is -1 then GetAppId 

• If this is the first access (ie *pAccessTokenId = - 1 ) then CheckoutLicense 

• If this is a renewal request: RefreshLicense 

• If there is a failure and it is due to eviction: GetEvictionReason 

Errors: 

#defme RPCRJJSER_AUTH_F AILED 

#define RPCR_ACCESS_TOKEN_INVALID 

#defme RPCR_ACCESS_TOKEN_EXPIRED 

#define RPCR_LICENSE_NOT_AVAILABLE 

#define RPCRJLICENSE_ALREADY_HELD 

#defme RPCR_EVICTION_NOTICE 



IN 


Subscription^ 


IN/OUT 


pAccessTokenld 


IN 


UserName 


IN 


PassWord 


OUT 


pStartTime 


OUT 


pStopTime 


IN/OUT 


Applicationld 


OUT 


PvPCReturnCodes 
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#defme RPCR_EVICTIONJVfUSTJJPGRADE 
#defme RPCREVICTIONENDMEMBERSHIP 
#defme RPCR_EVICTIONNO_PAYMENT 



DBReleaseAccessToken 

DBReleaseAccessToken(long AccessTokenld) 

IN AccessTokenld 
Processing: 

• Update the Usage table with the appropriate information. 

• Delete the LicenseUsage record. 
Notes: 

• We need a mechanism to release un-released access tokens. The way to do this 
would be to run a stored procedure at demand and at a predefined intervals to do 
this cleaning up. 

EvictAccessToken 

DBReleaseAccessToken(long AccessTokenld) 
IN AccessTokenld 

Processing: 

• Evicts an access token. 



Billing Component 

AddUsageRecord. Called by the SLM server when it releases an access token. 

Bool AddUsageRecord(String username, int subscription^, date starttime, long 
duration). 

GetUsageRecordsForUser. Used by external billing system. 

Couple[][] GetUsageRecordsForUser(String username) 
GetUsageRecordsForGroup Used by external billing system. 

Couple[][]GetUsageRecordsForGroup (String groupName) 
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Application Management Component 
AddApplication 

int AddApplication(String appname, String version, String description) 
Inputs: 

Appaname: Application name. 
Appversion: Application version 
Description. Application description. 
Outputs: 

-1 for failure to add the application. 
>0 otherwise. Application ID. 
Comments: 

Returns an app id for a newly added application. 

Errors: 

APPLICATION EXISTS 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetApplicationld 

int GetApplicationId(String appname, int version) 

Inputs: 

Appaname: Application name. 
Appversion: Application version 
Outputs: 

-1 for failure to find the application. 
>0 otherwise. 
Comments: 

Returns an app id for a given application. 

Errors: 

NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 

GetApplicationld 

int GetApplicationId(int Subscriptionld) 

Inputs: 

Subscriptionld 
Outputs: 

0 for failure to find the application. 
Omnishift Technologies, Inc. 2 1 Company Confidential 



eStream Web Server/Database Low Level Design 



>0 otherwise. 
Comments: 

Returns an app id for a given application. 

Errors: 

NO DATABASE CONNECTION 
UNKNOWN SQL ERROR 



GetSubscribedApplicationlds 

Int[]* GetSubscribedApplicationId(String username) 
Inputs: 

Username: username. 
Outputs: 

Array of application ids subscribed by a user. 
Comments: 

Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



GetUnsubscribedAppIicationlds 

Int[]* GetUnsubscribedApplicationId(String username) 
Inputs: 

Username: username. 
Outputs: 

Array of application ids not subscribed by a user. 
Comments: 

Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 

GetApplicationDetail 

CoupleQ GetApplicationDetail(int appid) 
Inputs: 

Application Id. 
Outputs: 
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Array of couple for the app id containing: 

{appname, appversion, description} values. 

Comments: 
Errors: 

USER NOT FOUND 

NO DATABASE CONNECTION 

UNKNOWN SQL ERROR 



Component design 

We will discuss some complex scenarios in this section. 
Subscription 

Single New User 

1 . Create the user. CreateUser. 

a. If a user already exists, return error message and go back to 1 . 

2. Create the account for the user. CreateAccount 

a. Get the contact information from the user. 

b. Prompt to get the billing information. The user may decide to not give the 
billing information at this point. 

Corporate group admin creating an account 

1 . Create the admin user. CreateUser. 

2. Create the group. CreateGroup 

3. Create the account information for the group. CreateAccount. 

c. Get the contact information from the user. 

d. Prompt to get the billing information. 

4. Add users to the group. AddUserToGroup. 

a. This method will automatically create the user if they do not already exist 
in the system. 

b. The list of users is accessible to the Group Admin by querying: 

i. Our database GetUserRecords OR 

ii. Some external database. Eg. LDAP directory. 



Single User subscribing to an application 
1 . Validate the user. ValidateUser 
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2. Prompt to get the billing information if the billing information is not already pre- 
sent. 

3. Get the list of un-subscribed applications. GetUnsubscribedApplications. 

a. GetUnsubscribedApplicationlds. 

b. For each app id returned, get the application details. GetApplicationDe- 
tail 

4. For each additional application user wants to subscribe, call AddSubscription 
SUM server checking out an access token to use an application 

1. Call DB Acquire AccessToken. 

Releasing unreleased access tokens. 

Unreleased access tokens will be release using a periodic stored procedure. The algorithm 
for stored procedure would be as follows: 

1 . For all records in the LicenseUsage table whose renewal time is past, call DBRe- 
leaseLicense. (Note that this can be a thread in SlimServer). 



UI Rules 

There are in general four kinds of UI components: textboxes, selects, checkboxes and ra- 
dio buttons. Certain client side validations need to be applied to the entries of these com- 
ponents for each of the interfaces we develop in order to facilitate server development 
and processing. These validations are detailed as follows: 

Textboxes 

• String literal — this kind of textbox entry can only contain letters, numeric digits, 
underscores and spaces. 

• Non-negative integer — this kind of textbox entry can only contain non-negative 
integers. 

• IP address — this kind of textbox entry will consist of 4 textboxes, each with a 
char width of 3, and each of them must contain between 1 and 3 numeric digits 
(inclusive). 

• Phone number — for the time being, we only care about US phone numbers. It 
will consist of 3 textboxes, the first two (char width 3) must contain 3 numeric 
digits each, and the last one (char width 4) must contain 4 numeric digits. 

» Password — this kind of textbox entry must be at least 5 characters long. 

Please note that they cannot be blank if they are required. 
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Selects 

• Drop-down — at least one selection must have been made if required. 

• List — at least one selection must have been made if required. 

• D fl ~ 5 is Wil1 COnsist of 5 ^op-down boxes, a selection must be made in each 
of them if a date is required, the default will be set to the current date and time on 
page load or reset. 

Checkboxes 

• At least one of a group must have been selected if required. 
Radio Buttons 

• One and only one of a group must have been selected if required. 



Testing design 

This document must have a discussion of how the component is to be tested Some sub- 
sections could include: 

Unit testing plans 

The follwowing components will be unit tested: 

ODBC connectivity dll for the SliM server and the Monitor. A simple C++ executable 
will be provided to test the SLiM server and the Monitor interfaces. The C++ executable 

• establish connection to the database. 

• simulate access token calls. 

• simulate the monitor calls. 

The Web servers' servlets and the JSP pages interact with the database using a set of java 
beans. Each of the bean will have a test interface. A simple JSP will be provided which 
will call all of the test interfaces for the beans. The test interface itself will be responsible 
to call all the interfaces that the bean provides in a predefined calling sequence. 

The servlets will be unit tested using a set of html forms which will invoke the servlets. 
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Stress testing plans 

Stress testing will be invoked using some external testing tool which can record HTTP 
traffic and replay the traffic for multiple users. Performix and LoadRunner are two possi- 
ble choices. (There may be additional tools available). 

Coverage testing plans 

Coverage on the ODBC components will use the same mechanism as the rest of the 
server code. Pure Coverage may be used to achieve this goal. 

Coverage on the Java components is an open issue. We need to investigate the appropri- 
ate tools for doing this testing. 

Cross-component testing plans 

The database is the central point for distribution of the data from Web server to the rest of 
die servers. Thus, creating the database data which can be used by Slim server and the 
App server will be a good source of cross component testing pis 



lan. 



Upgrading/Supportability/Deployment design 

We will be finally shipping just the java class files and JSP pages to the customer It is 
assumed that the customer will have the appropriate web server to support JSP 1 1 and 
Servlet2.2. 



Open Issues 

l. 

etc. 



We have assumed that the JDBC implementation will come from Inet software 
We may need to change to an alternate JDBC vendor based on pricing, quality 
etc. 

2. Tomcat is decided to be the JSP/Servlet engine. Again, this is a freeware and may 
be replaced by a commercial version (Jrun from Allaire) 

3. The web server will need to talk to the Monitor. The messaging component for 
this communication is not well defined yet. 



Omnishift Technologies, Inc. 



26 



Company Confidential 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ image CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 



U REFERENCE(S) OR EXHIBIT(S) SUBMITTED ARE POOR QUALITY 
□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




COLOR OR BLACK AND WHITE PHOTOGRAPHS 



□ GRAY SCALE DOCUMENTS 

J 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 




